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Preface 
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owned by the two organizations. The Consortium will receive a share of the proceeds 
from sale of the Guide io support its research and development efforts. Member states 
will also benefit through significant discounts on the cost of the Guide. 

The Consortium is a coalition of guidance representatives from participating state and 
territorial departments of education dedicated to enhancing career guidance and 
counseling, leadership, research, and dissemination in education. The Consortium's 
goal is to provide a framework for improving the effectiveness of elementary, secondary, 
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"Which ivay do I go?" said Alice. "That depends a good deal on where you want to get 
to, " said the cat. 7 don't much care where, M said Alice. Then it doesn't matter which 
way you go, " the cat. "So long as I get somewhere, " d/ta? adirietf as aw explanation. 

(Lewis Carroll, /l/Zte im Wonderland) 



Purpose of the Guide 

Evaluation is an integral part of every program and when considered during the 
program development stage, assures clear, measurable goals. In guidance, traditional 
services frequently have been evaluated after the fact, without the benefit of clear goals 
being established from the beginning. Therefore, counselors have found themselves in 
the situation of being evaluated by students, parents, administrators, and community 
members based on individual perceptions of "what the counselor is supposed to do" 
rather than on established criteria for predetermined student results. Traditionally, 
counselors have defined their jobs in terms of the processes they use and then have 
been evaluated by counting the number of students utilizing the services offered. 
Metaphorically, it is akin to shooting an arrow at the wall and then drawing a target 
around it rather than drawing a clear target, shooting at the target, and then measuring 
how close one came to hitting the bull's-eye. 

It is often said in education, as well as in other fields, that what gets measured gets 
done. Current trends in education which ftxus on results are identified as outcome 
based education, competency-based education, and results-based education. Similarly, a 
growing number of guidance professionals have aligned their guidance programs with 
the results-based trends in education. After being torn between defining themselves as 
mental health professionals, "mini-shrinks," or administrative aides, guidance 
professionals arc now clearly identifying themselves as educators, responsible for 
teaching students specified knowledge, attitudes, and skills in guidance related areas 
such as learning to-learn, learning to -work, and learning to-relate effectively to self and 
others. When seen in this light, the counselor becomes an important part of the 
evaluation process and uses evaluation data on a daily basis to improve the program in 
a heuristic manner. 

Often literature on program evaluation seems to fix*us on information written by 
researchers and (."valuators for other researchers and evaluators. Many districts establish 
departments of evaluation that design guidance program evaluation strategies, often 
with little input or involvement by counselors. This misconstrued perception of 
evaluation as being primarily external to the program has been especially hurtful to 
guidance programs. The service or process nature of traditional guidance, compounded 
by counselors' not having specific content objectives, has caused evaluation efforts to 
focus on subjective and somewhat uncontrollable results, such as, "Do you like your 
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counselor? What is your counselor's name? Was your counselor helpftil? How often did 
you see your counselor? 1 * Given the fact that students are often referred to guidance for 
purposes not valued by the student (behavior problems, class changes, parent 
conferences, or poor grades), evaluation has become a feared and avoided function. 

This Guide was developed by guidance professionals interested in including 
counselors in evaluation efforts. Traditionally, the language and purpose of evaluation 
and the appropriate use of findings have not necessarily been applicable for counselors 1 
use. However, evaluation can and should provide counselors with essential information 
for decision making, problem solving, resource allocation, modification of processes, 
and communication with other decision makers such as administrators, school board 
members, and community groups. 



Benefits of Evaluation 



Evaluation is only beneficial if the data collected are related to the decisions being 
considered. Evaluation is the collection and reporting of information that is useful, in 
decision making, to see whether or not results have been attained, and to ascertain the 
effects of a program. For the purposes of this Guide, evaluation is defined as "a 
process designed to provide data comparing results with purposes/* 

Evaluation seeks to answer the following questions (Kaufman, 1980): 

1. What are the goals of the organization? 5. How well is the program administered 
(Do the guidance goals reflect the and managed? 

mission, philosophy, and goals of the 6. What, if anything, about the program 
school board?) should be changed? 

2. What results are intended by the 7. what, if anything, about the program 
program? should be continued? 

3. What results are obtained by the 8. Should the program exist at all? 
program? 

4. What are the value and usefulness of 
the methods and means used to 
achieve the results? 

When evaluation is viewed as a judgmental process in which the results are used in 
hurtful or punitive ways, it ceases to be a useftil program development and 
improvement tool. When it is seen as an ongoing, changing, dynamic part of the 
guidance program, however, it is invaluable to the growth and improvement of the 
program. The counselors 1 understanding of the process, their involvement in the 
development of evaluation strategies, their inclusion in program improvement 
decisions, and their ability to use evaluation data in their daily as well as long range 
planning enables evaluation to take on an increasingly significant role in the field of 
guidance. 
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Likewise, the participation of administrators, staff, parents, and community in evaluation 
design, data collection, and decision making can serve to strengthen the sense of 
commitment and interest of others in the guidance program and its results. If the 
purpose of the guidance program is for all students to gain the specified guidance 
competencies, then it is incumbent upon counselors to involve other adults who affect 
a student's development (teachers, parents, and community members) in helping to 
provide and reinforce the guidance competencies. Participation of others in the 
evaluation process makes the goals of the program visible and understandable to 
others. Of course, there are also dangers in involving non-educators and non-guidance 
professionals. By observing some caution in the scope of involvement, however, and by 
clearly defining the parameters of each group's involvement, participation by others can 
provide a more diverse and comprehensive overview of the program results. 



Assumptions 

An attempt has been made to start from ground zero, so that counselors and others who 
have had little experience in program evaluation can follow the process step by step, 
with resources identified from districts throughout the nation. It is recognized that there 
is no one accepted model of guidance, and therefore the authors have presented 
evaluation ideas that can be adapted for different models. Also, evaluation samples have 
been drawn from a variety of program perspectives. 

There is a need in guidance to go beyond the traditional pre/post research-based 
evaluation models and to provide results-based models that are useful to counselors. 
There is also a need to provide counselors with resources they can use in helping 
others avoid evaluating their guidance program based on inappropriate data such as 
standardized test scores, college admittance to "prestigious" schools, and number of 
scholarships awarded. Evaluation uses data collection and measurement to compare the 
data against an established standard and to identify gaps or discrepancies between 
current results and desired results in order to: (1) determine what should be done to 
close the gaps and (2) assure that counselors can pick the best means to achieve the 
desired results (Kaufman, 1980). Therefore, evaluation findings focus on what works to 
reach the intended goal (successes) rather than on rating individual counselor 
effectiveness or the level of student participation in guidance processes. 
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Possible Obstacles 



In preparing this Guide, the authors listened to the concerns of counselors, 
supervisors, and state level guidance professionals in an effort to address the problems 
perceived by practitioners. The following constraints were mentioned as obstacles to be 
addressed: 

1. Most districts do not have a budget for counselors to evaluate their programs. 
Therefore, many guidance ev aluations are done by a department of evaluation and 
research or by outside evaluators. Consequently, many school-based staff members 
have come to believe that evaluation is external to as well as irrelevant to the daily 
work done in schools. Although counselors may be called on to participate in 
evaluation projects, these efforts are often perceived as the part of the job called 
"other duties as assigned." 

It is the purpose of this Guide to provide evaluation strategies that are an 
integral part of the ongoing guidance program, needing no additional 
funding or resources. As counselors assume ownership for data collection 
as part of their guidance plan, evaluation becomes a self-correcting 
process that Is both formative and summative in nature and relates 
directly to the counselor's daily activities. 

2. revaluation findings have frequently been used to criticize guidance counselors and 
have focused on negati v e results. Therefore, counselors and other educators have 
grown to fear evaluation. 

When evaluation data are used as part of the counselor's planning process, 
the counselor gains control of the ability to address needs, that is, 
discrepancies in results, as gaps become obvious. In this way, there are no 
surprises and the program can be adapted at any time to address specific 
concerns. This approach reduces the opportunity for others to use 
evaluation "against" counselors, since counselors are part of the 
evaluation design and subsequent decision making. "The terms 
evaluation, testing, and measurement sound depersonalizing, and some 
educators infer that people are being separated f<*om their humanity, 
competence, caring, and sensitivity. Good evaluation does not confuse 
those concepts with blaming and buck-passing" (Kaufman, 1980). 

3. There is confusion between tests and evaluation. Tests are seen as causing more 
paperwork with little return for the counselor. 

"Evaluation uses testing (data collecting), measurement (the method of 
applying standards to behaviors), and evaluation to place values and 
worth among alternative activities, events, and results" (Kaufman, 1980). 
Counselors must be willing to move beyond determining only the extent 
and worth of what is and relate evaluation to useful change. When used in 
this manner, evaluation is more than comparing that which was intended 
and moves to consideration of the new, the useful, and the creative in 
terms of possible purposes and needs. Counselors welcome the 
continuing flow of data as a means of increasing their personal 
effectiveness and creativity. 
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4. There is a fear that evaluation will be used statewide or districtwide to expose 
differences among schools, which will be perceived by the public as lack of 
effectiveness. 

Reading, understanding, and acting upon the results of an evaluation are 
equally important in the evaluation process. The evaluation report 
becomes an important tool in defining how the results are perceived and 
used. Chapter Five of this manual addresses the many issues involved in 
using and reporting data. The misuse of data is an ongoing problem in 
education and is currently being addressed on many fronts, including the 
testing companies, such as American College Testing and Educational 
Testing Services, which have been leaders in identifying appropriate and 
inappropriate use of testing data. 



Audience 

The audience for this Guide includes program planners, program implemented, and 
recipients of guidance evaluation reports. School counselors, managers of guidance at 
the district, county, and state levels, decision makers and opinion leaders, and the 
general public all have a stake in assuring that students benefit from the guidance 
program. It is expected that the practical step by-step suggestions, the different 
perspective on guidance evaluation, and the cautions relating to the inappropriate use 
of evaluation data will serve all the audiences of this manual. 



How This Guide Can Be Used 

This Guide can provide a comprehensive program evaluation or can be used to 
evaluate only selected parts of a program. It is recognized that some schools or school 
districts will have a results-based guidance program in place that is ready for evaluation, 
while others may need a year or more before the basic program elements are designed 
and implemented for a results-based orientation. Therefore, this program manual is 
recommended for: 

1. Total program evaluation only if all program elements are in place and ready for 
evaluation 

2. Evaluation of a specific aspect or element of a program such as: 

a. evaluability (whether or not the program is ready to be evaluated: a program 
audit is presented to help with this determination) 

b. guidance resources and guidance pnxesses (means used to obtain results, which 
may include staff, time, strategies, facilities, materials) 

c. student competencies (short-range results, en route indicators) 

d. student terminal competencies/goals (at time of leaving school) 

e. long-range outcomes ( follow-up studies, impact three, five, ten years after 
graduation) 
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3. Addressing specific evaluation questions/issues, such as what kind of additional 
resources are needed to ensure that all students attain the desired competencies 

4. A cookbook of evaluation strategies, tools, techniques, and examples. 



Content 



Chapter 1 is an introduction to evaluation 

• Definition and purpose of program 
evaluation 

• Differences between research and 
evaluation 

• Differences between results based and 
services-based guidance models 

• Description of two different results- 
based models 



concepts, including: 

• Identification of what is evaluated 
(content of a guidance program) 

• Similarities between evaluation of 
guidance and the evaluation of other 
educational programs 

• Relationship of evaluation terminology 
and program elements 



Chapter 2 introduces the elements common to results-based guidance models. 



Standards for determining whether a 
program is results-based 
Linkages between district purpose/ 
mission and guidance program 
Assessing three kinds of results 
Identification of system elements and 
criteria for each clement 



Putting it all together in a systemic 
results-based guidance program 
Description of differences between 
various results based guidance 
programs 



Chapter 3 examines the readiness of a guidance program for results evaluation. 

Every guidance program must contain hasic program elements before it can be 
considered ready to be evaluated. While not every program will have all of the 
elements, without the requisite elements only limited aspects can be evaluated, and the 
findings and recommendations may not reflect the actual needs of the students, staff, or 
program. 



When is guidance a program? 

What are the basic elements of a 
guidance program? 

- Program structure, processes, and 
activities 

- Program content 
Program resources 



How are identified standards used and 
a program audit completed? 
How are time and task distribution 
analyzed? 
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Chapter 4 provides criteria for determining whether a guidance program is results 
based for the purposes of program evaluation. It also addresses assessment strategies for 
three kinds of student results (immediate, intermediate, and long-range), with 
examples of how to assess each. 



• Introduction of the elements of a 
results-based program 

• Linkages Detween district, school, and 
guidance program purpose 

• Assessing results 

• Collecting data for planned results and 
unplanned-for results 



Putting it all together in a plan 
Program goals and criteria for success 
Evaluation strategies for long-range 
results 

Counselor accountability for results 



Chapter 5 addresses the purposes and uses of both process and product evaluation 
data. 

• Types of evaluation 

• Types of data collected 

• Uses of evaluation 

• Decisions and actions 



Audiences 

Evaluation reports: Content, format, and 
methods 



Chapter 6 describes the process of developing an evaluation plan in a step by step 
format. The planning process, forming the questions to be addressed, and seeking 
answers by analysis of data are included. 



• The way that different evaluation 
models organize the evaluation 
elements 

• Differences between process and 
results evaluation 

• Questions to be addressed according 
to the program area being evaluated 

• Sample evaluation questions for use 
with a variety of participants, sampling 
different areas covered within the 
guidance program 



Variety of ways to collect data and when 
to use each 

Step by-step plan for implementing a 
guidance evaluation 
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Epilogue and Appendices 



The epilogue and appendices have been arranged to provide some specific examples of 
evaluation elements. 

Epilogue — The person-oriented approach in evaluating career guidance programs (an 
example using specific instruments and processes) 

Appendix A — Needs assessment models (descriptions of various types of needs data 
and instruments) 

Appendix B— References for evaluation resources 



Diversity of Viewpoints 



This Guide was developed over a three year period with input from a number of 
experts and practitioners in the field of guidance and counseling. All the contributors 
met several times to discuss evaluation of guidance programs. Although each had a 
different idea about how best to develop a results-based guidance program, there was 
consensus on the need to evaluate the effectiveness of guidance in terms of student 
outcomes. 

The diversity of viewpoints presented in this book reflects the many different kinds of 
results-based programs found in school systems. To prevent confusion, an effort has 
been made to use consistent terminology throughout. In cases where authors use 
different terms for similar elements, all other terms used in this Guide have been 
included in parentheses immediately following. In addition, the definitions of similar 
terms have been included in Chapter 1. 



References 
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CHAPTER 



ONE 



What Is 

Guidance Program Evaluation? 

Sharon K. Johnson, California State University, Los Angeles 



After completing this chapter, ycu should be able to: 

• define guidance program evaluation. 

• define the purpose of guidance program evaluation. 

• explain the difference between research and 
evaluation. 

• identify the differences between results and services- 
based guidance models, 

• differentiate between guidance means and ends for 
evaluation purposes. 

• describe similarities and differences between two 
different approaches to results-based guidance 
programs. 

• recognize the similarities between guidance program 
evaluation and evaluation of other educational 
programs. 

• recognize specific evaluation terms as they relate to 
guidance program elements. 

• identify the elements common to all results-based 
guidance programs. 
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"Faith, hope, and charity have characterized the America n attitude toward guidance 
programs— faith in their effectiveness, hope that they can meet important if not 
always clearly specified need, and charity in not demanding more evaluative evidence 
that the faith and hope are justified. " 

(Tamminen and Miller, 1968, 3) 

To address the definition and purpose of program evaluation, one must first understand 
the purpose of the program being evaluated, as well as rhe purpose of evaluation. To 
evaluate guidance results, the guidance program must be defined in terms of the 
desired student results rather than in terms of the processes/activities provided by the 
guidance program. For the purposes of this Guide, it is assumed that the defined 
purpose of the guidance program is to ensure that all students gain competencies in the 
areas of learning, working, and relating to self and others. In other words, the guidance 
program is designed to provide a framework which enables a variety of professionals 
within the school working together to ensure that all students have the knowledge, 
attitudes, and skills that will allow them to become successful while they are in school 
and later in their post-high school endeavors. Evaluation identifies the extent to which 
students have gained the expected competencies. 



Definition and Purpose of Program Evaluation 

Evaluation is the collection and reporting of information that is useful in decision 
making. It is a process designed to provide data for management decisions while the 
activities are being implemented as well as at the end of the activities. 

The primary purpose of program evaluation is to provide guidance workers with 
evidence of their contributions (achievements/successes) that will then allow them to 
focus efforts on improving an activity or set of activities. Evaluation can also be used to 
show that a specific goal has been reached. The primary purpose for collecting 
information is to guide future actions to improve future results. In short, "The purpose 
of evaluation is . . . to improve" (Stufflebeam, Foley, Gephart, Guba, Hammond, 
Merriman, and Provus, 1971). 

Evaluation is not synonymous with research. Evaluation is a judgmental process that 
requires an assignment of value to the data in informed management decision- making 
activities. "Decision making is a subjective act in which value judgments are applied to 
evidence produced by evaluation . . . But the decision makers, not the evidence, make 
the decision" (Mitchell, 1984, 2). Evaluation considers a variety of types of relevant data 
or information in determining progress toward a goal or worth of a program. "In 
educational evaluation, all available types of data, qualitative as well as quantitative, may 
be brought to bear" (Noll and Scannell, 1972, 15). Evaluation data must be not only 
accurate but also related to the appropriate purpose. 
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On the other hand, research most often ftx-uses on whether one process is more 
effective than another pnxess and whether either is more effective than no process in 
reaching a predetermined goal or objective. This method is ineffective and, in some 
cases, unethical when working with students to help meet their goals, Counselors will 
continually change processes during their work with individual students or groups if 
they find what they are doing is ineffective or if another process appears to better meet 
the immediate needs of clients. It is the heuristic nature of the guidance process that 
presents difficulty with traditional experimental research methods. 



Differences Between Results-Based and Services-Based 
Guidance Models 

There are clear differences between results-based and services-based guidance models. 
Results based programs identify the student as the primary client and define the 
program in terms of the specific competencies that each student will achieve. Services- 
based models focus on the identification of the specific services/processes available for 
students, such as counseling, placement, and coordination. Results-based programs are 
planned and implemented for all students; services-based models are planned and 
implemented to provide an opportunity for students to benefit if they choose, request, 
or are required to use a service. Results-based programs are evaluated on the 
competencies students acquire, services-based models are usually evaluated on how 
many students used the services and how they felt about the services. 

Results-based programs are usually proactive; services-based models are reactive. In 
order to reach all students, a results-based pnjgram must utilize a comprehensive, 
systematic, sequential, and flexible approach; a services-based approach offers specific 
processes that are constant. Results-based programs utilize and appreciate individual 
strengths and interests of professionals; services based models arc characterized by all 
professionals following similar "roles and functions" and being appreciated for doing 
the same as their colleagues in an efficient and timely manner. 

This Guide provides guidelines for determining the effects that a results based 
guidance program has on all students. There has been no attempt to include 
information on how to evaluate a services-based guidance model. 



Two Different Results-Based Models 

Results-based guidance programs have evolved from the belief that students need a 
comprehensive developmental guidance program. The difference between results- 
based guidance and guidance services is a basic philosophical difference between 
offering students an opportunity to experience and benefit from guidance at their own 
request and providing a planned, sequential program in which counselors take 
responsibility for assuring that all students gain specific guidance-related competencies 
( knowledge, attitudes, skills). 

12 
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There are several rruxlels and approaches to results-based guidance, which have 
definite similarities and differences. Two different models are presented, with different 
perspectives but with agreement on the importance of student attainment of specified 
guidance-related outcomes as the primary focus of a planned, sequential guidance 
program. 

The differences between the two results based guidance programs presented in this 
document relate to the management philosophies underlying each. The first results- 
based program (Chapter 2), called "competency based guidance," argues that if the 
desired result remains constant, the process cannot remain constant. In order to achieve 
the desired program results for all students, a variety of pnxesses arc used to 
accommodate individual staff differences ( interests, expertise, etc.), student differences 
(learning style, developmental stage, needs, etc.), and resources. If the process is held 
constant, program results will necessarily vary. In this model, the results are established 
as a given for the program, but the specific contributions of individual counselors are 
agreements forged between counselor and administrator. The agreements on 
contributions arid the plans to achieve the agreed upon results must be flexible and 
must change as the needs change. 

The second program (Chapter 3), called "time on task," addresses the need for 
counselors to spend all their time on guidance-related presses. Therefore, an attempt 
is made to define a balanced program which includes specific pnxesses (curriculum, 
classroom units, group and individual counseling) and mandated time allegations for 
each. The definition of counselor contributions and plans is defined as an integral part 
of the guidance program. It is also results based because it has defined student 
outcomes as the basis for program evaluation. Although there is more definition of 
processes to be used in the delivery 7 of student results, it is the attainment of student 
results that shapes the program. 



What Is Evaluated 

Guidance has a body of knowledge (content) within the educational, career, personal, 
and social domains. Through the guidance program students acquire competencies in 
learning how to learn (knowledge of self as a lifelong learner and the skills to be a 
learner), learning about the world of work and gaining skills in becoming a worker, 
learning to relate to others, and developing good mental health practices. The progress 
of students in gaining knowledge, attitudes, and skills in the guidance-related content 
areas is the focus of results-based evaluation. 

When the targets (student competencies) are defined ahead of time, the evaluation 
becomes a means of measuring how close the program comes to reaching the target, in 
order to provide for correction in the prcx-esses for the purpose of achieving total 
success or improving the program. Conversely, when the target has not been clearly 
defined, the evaluation becomes a prexess of "bean counting," that is, counting how 
many students experienced each guidance activity irrespective of the results attained. 
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Judging program effectiveness on the basis of student participation is ineffective 
because there may be little relationship between participation and competencies 
attained. This becomes evident if we equate guidance program outcomes with the 
outcomes of other curriculum programs; i.e., the number of students taking a particular 
class does not necessarily correlate with the number of students mastering the skills 
being taught. 

The specifics of u } hat gets measured are related to when results are measured. 
Evaluation takes place concurrent with the program activities as well as at the end of 
activities, but what gets measured may be different. 

1. During program implementation, specific student knowledge, sk'Us, and attitudes 
are assessed in much the same way they would be assessed in a regular classroom 
at the end of a lesson or activity. This evaluation is an ongoing part of the guidance 
program data collected on a regular basis, compiled, and reported periodically for 
accountability reasons. 

2. At the time the student leaves school, the goals (which are a composite of 
competencies taught throughout the program ) should be assessed. Indicators may 
include completion of high school requirements, acceptance into a post-high 
school program of choice, employment, and development of realistic post-high 
school career and educational plans. Other individual competencies may include 
evidence of decision making skills, interpersonal skills, study skills, knowledge of 
personal aptitudes and interests, knowledge of related occupations, etc. 

3. Three to five to ten years after graduation, outcome or impact assessment focuses 
on whether the student has assumed a productive role in society as a citizen, an 
employed worker, and a contributing member of a community, a family, and 
scxiety. 



Similarities Between the Evaluation of Guidance and the 
Evaluation of Other Educational Programs 

Recently, much has been written about outcome based education related to a number 
of different curriculum areas. Guidance has a curriculum (content) that can be 
evaluated much as other curriculum areas are evaluated. How the students feel about 
the skills attained is less important than what the students have learned or the skills 
they have acquired. Although attitudes and opinions are important within the guidance 
program, often students cannot judge the importance or relevance of competencies at 
the time they are being learned; however, they may be critically important to the 
students' success in school and later on. 

It is not necessary that standardized instruments or quantitative data become the only 
evidence seen as valid in the measurement of guidance competencies. It is important 
that more than one type of measurement be used to determine program effectiveness. 
As in other curriculum areas, a test may be used to determine a student's attainment, 
but qualitative contributions such as class discussions, homework, and projects are also 
used and valued. 
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Long range program results are best measured by examining evidence of demonstrated 
competencies as they are used in life decisions. Long-range results are the cumulative 
effect of all of the student's life experiences and, therefore, cannot be directly and solely 
attributed to the guidance program or to the educational system. However, evaluating 
the successes and failures of graduates gives us a clue to program strengths and 
deficiencies that may make a difference in students' lives. 



Relationship of Evaluation Terminology to Program Elements 

Figures 1.1, 1.2, and 1.3 (C. D.Johnson and Sharon K.Johnson, 1983) indicate the 
relationship between many of the evaluation terms used in education and the elements 
of a guidance program. What is evaluated, who should do the evaluation, when 
evaluation is done, who is evaluated, and suggested strategies are listed to provide a 
framework for evaluation terminology. 



MEANS (PROCESSES) 


ENDS (RESULTS) 


INPUTS 


PROCESSES 


PRODUCTS 


OUTPUTS 


OUTCOMES 


• Counselors 


• Counseling 


• Student 


• Goals 


• Philosophy 




Competencies 






• Tests and 


• Group 


• Learner 


• Summative 


• Purpose 


Scores 


Guidance 


Outcomes 


Data 




• Career 


• Planning 


• Learner 


• Terminal 


• Impact 


Information 


Objectives 


Performances 




• College 


• Management 


• Formative 


• Graduation 


• Summative 


Catalogues 


Activities 


Data 


Completion 




• Film Strips 


• Assemblies 


• En-route 


• Completed 


• Societal 




Indicators 


Training 


Consequencs 


• Time 


• Parent 










Conferences 










• Staff 










Development 










• Referrals 










• Homework 









Figure 1.1 

Relationships ol Evaluation Terminology 
With Organizational Elements ftdodel 



1. Inputs, resources. These terms refer to the basic ingredients of a guidance 
program— the people, facilities, time, materials, and supplies 

2. Processes, activities. The "hows" of guidance program implementation. These 
strategies are designed and used in a flexible manner to help reach the desired 
student results. Processes will vary according to the student populations (the success 
of specific processes used with regular students may differ from those used with 
Special Education students or with fifth graders vs. seniors in high school); the 
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individual counselor's competencies (some counselors are more successful with 
large group than with small group counseling, while others prefer working with 
individual students); the available resources (some schools may have 
comprehensive career centers, while others have only small resource corners in the 
school library); or other conditions that affect the program. 

3. Results, outcomes, products, goals, en-route indicators, impact. These refer 
to the attainment of competencies as a result of a guidance program activity or 
pnx'ess. Guidance competencies are attained throughout the student's educational 
career. However, immediate competencies will differ from terminal (cumulative) 
competencies. For example, knowledge of one's educational strengths and 
weaknesses is an immediate knowledge competency, which, when combined with 
other competencies will be used in the development of a life career plan (;j 
terminal competency). 

4. Program audit, means evaluation. A program audit is used to determine that a 
guidance program exists. A program cannot be evaluated until certain criteria are 
met (sec Chapter 3). The program audit determines that appropriate means 
(resources and pnxesscs ) are in place. If there are no coun selors, no materials, or 
no guidance program activ ities, there is no guidance program to be evaluated. 
Contrary to the traditional perception of guidance services, hiring counselors to sit in 
offices to see students on a first come, first-serve basis is not a guidance program and 
therefore cannot be evaluated for student results. 





Inputs 


Processes 


Products 


Outputs 


Outcomes 


What 


• Documented 
Time 

• Funds for 
Materials, 
Persons, etc. 

• Resources such 
as Persons and 
Materials 


• How Many 
Went Through 
A Process or 
Activity 


• Continual 
Verification 

• Educational Plan 

• Friends 

• Contributions to 
School, Home, 
Community 


• College 
Acceptance 

• Employment 

• Advanced Training 
Plans 


• Financial 
Independence 

• Productive 
Community Roles 

• Positive 
Relationships with 
Others 


How 


• Audits 


• Sign-in sheets 

• Roll sheets 


• Results of 
Paper/ Pencil 
Inventories 

• Discussions 

• Observations 


• Review and 
Validate 
Documents 

• Interviews 


• 1,3,5, 10-Year 
Follow-up by Mail, 
Phone, Personal 
Interviews 


Who 


• Counselors 

• Administrators 


• Counselors 


• Counselor 

• Parent 

• Teacher 

• Peer 


• Persons Internal to 
the School or 
System 

• Persons External to 
the School or 
System 

• Employees 

• College Admissions 
Officers 


• Persons External to 
the School or 
System 

• Persons Internal to 
the School or 
System 

• Community 

• Society 



Figure 1.2 
Alternative Evaluation Strategies: 
The What, The How, and The Who 
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Resources 

• Inputs 


Processes 

s Activities 
s Services 


Immediate Results 

s Student Competencies 
s Products 
• Learner Objectives 
s Micro-Results 


Intermediate Results 

s Goals 

s Outputs 

s Macro-Results 


1 Ann.rannit QaaiiIIa 

s Purpose 
• Outcomes 
s Mega-Results 








Competencies 


Impact 


Goal 


Impact 


Impact 


Who 


Administrators, 
counselors 


Secretary, parents, 
counselors, 
volunteers, 
learners 


Counselors, 

teachers. 

Career 

Technician, 

parents. 

learners 


Counselors, 
school 

administrator 


Counselors. 
Central Office 
personnel, 
learners 


Centcal Office 
personnel 


Central Office personnel 


How 


Increase/ 
decrease in 
money, persons 
involved. 

materials, cost per 
participant 


Counting persons 
going through a 
process or using a 
service: paper- 
pencil test asking 
perceptions of 
how a person 
feels about it— if 
s/he likes 
process/ service, 
was it helpful 


Observing, 
discussion, 
paper-pencil 
test, 

completing a 

learning or 

career activity 

packet 

recording 

verification. 

Career 

Planning 

Folder 


Changes in 

available data 

on drop-outs. 

attendance, 

referrals, 

program 

changes, 

failures. 

standardized 

tests, other 


Interview, 
check 
education & 
career plans, 
paper-pencil 
test, simulated 
demon- 
strations, 
portfolio 
review, other 


Use available 
ddta (dropouts, 
grades, test 
data. 

vandalism, 
referrals, 
juvenile crime, 
etc.) career 
plans: random 
sample 
through 
intervention 
methods, other 


Follow-ups. written 
surveys, data retrieval., 
phone calls, personal 
interviews, state and 
federal data bases, 
criminality, college 
completion, employment, 
advanced training, 
community involvement 
other 


Population 


Number of 
persons getting 
assistance 


Those 

participating in a 
process or using a 
service 


All students, 
staff and/ or 
parents 
participating in 
the activity 


Specific 
students' 
behavior 
changes for all 
or sub- 
populations 


Graduating 
seniors, drop- 
outs, program 
completers 


Graduating 
seniors, drop- 
outs, program 
completers 


Stratified random sample 
of graduating seniors 


When 

i 

i 

I- 


After activity has 
been completed 


During activity or 
at the end 


Immediately at 
the end of the 
activity 


Ongoing 


Last semester 
of school or 
end of 
program 


When time 
allows 


(Suggest) 1, 3, 5. and 10 
years after graduation 



Figure 1.3 
Who, How, Population, When 



5. Evaluation, research. Evaluation and research differ in terms of purpose, 
audience, results, and swial utility. Experimental research designs are impractical for 
ongoing program evaluation purposes and do little to help counselors and other 
decision makers determine the next steps in program improvement. 

6. Purpose, mission. The reason a program exists is defined in the mission and or 
purpose statement. It is the purpose/mission that is evaluated to determine long- 
range impact. Data on students one. three, five, and ten years after graduation tell 
whether the program accomplished what it was intended to accomplish. 

7. Philosophy, assumptions. The philosophy or assumptions define the guiding 
principles and guidelines for the program. Sometimes defined as a series of "We 
believe" or "We agree" statements, the philosophical tenets must he accepted by 
and adhered to by all practitioners. For example, a key concept for consideration in a 
results based guidance program is whether the program intends to guarantee results 
for all students. Before such a stance can be agreed upon, the implications for 
evaluation must be understood by all concerned. 
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8. Immediate, intermediate (terminal), long-range results. Immediate results 
may also be referred to as student competencies. The results of a given process or 
activity, intermediate or terminal results are determined at the time the student 
leaves or graduates from school. In some models, the terminal results are called 
competencies, in others, they are referred to as goals. Long-range results are 
determined three to five years after graduation and help to evaluate whether the 
program has met the intended purpose or mission. 

Although some of these terms are used differently in different programs, there is 
agreement on the need for: (1) an overall statement or statements of intentional ity 
(often called a purpose or a mission statement); (2) statements of the guiding 
principles or tenets of the program (philosophy); (3) specific goals or student 
outcomes intended for all students; (4) definition/description of guidance program 
content (may be in the form of a conceptual model); (5) activities/processes 
designed to help students attain the stated goals/competencies; (6) resources 
assigned by school administration for the express purpose of meeting guidance 
goals; and (7) needs assessment which may be done either deductively (prior to 
program development), or inductively (after desired program results are defined in 
order to determine needs within the defined program parameters). 
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CHAPTER 



TWO 



Results-Based Guidance Programs 

C. D. Johnson, Johnson and Associates 



After completing this chapter, you should be able to: 

• list the standards necessary to determine whether a 
guidance program qualifies as "results-based." 

• explain the necessary linkages between a district 
purpose/mission and the purpose of the district 
guidance program. 

• define three kinds of results in terms of what the student 
will learn. 

• list the system elements of a results-based guidance 
program and identify the criteria related to each 
element. 

• explain how the elements of a results-based program fit 
together to provide a comprehensive program of 
student results. 
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Results-based guidance programs provide answ ers to the questions "How do students 
benefit from guidance?" and "How are students different because of guidance or 
student support programs?" In results- or competency based programs, the student is 
the primary' client and the program is evaluated on the competencies (knowledge, 
skills, and attitudes') that the students acquire. 

A guidance program is results-based when it has < a J clearly delineated results 
(immediate, intermediate, and long range) and ( b ) means and resources to achieve the 
results. A guidance program is results- based when the following criteria (or standards) 
are in place: 

1. A statement of purpose or mission that defines the intention of the program in terms 
of its long-range impact on students and is directly linked with the board of 
education's statement of purpose or mission. 

2. Intermediate student results or program goaktlm indicate what students will know 
and or be able to do as a result of the guidance program by the time they graduate 
from the school district. 

3. Immediate results (or stttdent competencies) that define the knowledge, attitudes, or 
skills that students will acquire as a result of guidance program activities throughout 
their school career. 

4. A clear definition of the guidance content to be learned and assessed. This 
definition may be in the form of a glossary or a separate document and forms the 
curriculum that >s to be delivered through the instructional process in order to assist 
students to achieve and demonstrate the results described above. The processes 

( individual group counseling, guidance classes, teacher led units, special events, 
parent activities, etc.) will vary depending on the expected results, student readiness, 
staff competencies, available resources, and organizational policies and procedures. 

The three kinds of results (immediate, intermediate, and long-range) are defined in 
terms of what the student will learn and be able to do, not in terms of services the 
student received or the student's participation in guidance activities. 

The following set of elements (see Figure 2.1) and related criteria (Johnson and 
Johnson, 19S2) offer an example of a results-based guidance program. 



Elements of a Results-Based Guidance Program 

1.0 Statement of Purpose or Mission 

The statement of purpose or mission defines the intention of the program and indicates 
the student results three, five, or ten years after high school. The desired long range 
impact of the guidance program must directly link with the statement of purpose or 
mission of the Board of Education and administration. The statement of purpose 
defines why guidance exists in the schools and is the bottom line when assessing the 
success of a program. Are the students ultimately successful in the world as a result of 
their education? 
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1. PURPOSE 



10. 




7. COUNSELOR 
AGREEMENTS 



Figure 2.1 
System* Approach to Guidance 
©1985 by C. D. Johnson 



The criteria for the statement of purpose for a results-based program differ from those 
for a services-based program, which may indicate that the program is available to all 
students or that any student has an opportunity to benefit from the program. A results- 
based program speaks to the competencies that are to be acquired by all students, not 
just those who opt to take advantage of them. 

Criteria. The statement of purpose 

1.1. is written with the student as the primary client. 

1.2. is written for all students. 

1.?. indicates categories of content or skills to be learned. 

1.4. links with the statement of purpose or mission of the administration or board of 
education. 

2.0 Statement of Philosophy 

The philosophy is an articulated set of principles and beliefs that guide the student 
support or guidance staff in the development, implementation, management, and 
evaluation of the results-based program. To make this a usable vehicle within the 
program development efforts, consensus and clear understanding of each statement are 
necessary. For example, if the staff states that student progress will be monitored by the 
parent and student, there are important implications about how the professionals will 
be involved in the process. Therefore, the statement of philosophy becomes the 
foundation that supports and, in part, defines all other program elements. 

Criteria. The statement of philosophy 

2.1. addresses all students. 

2.2. focuses on primary prevention and student development. 

2.3. identifies persons to be involved in the delivery of activities. 
2A. identifies who will plan and manage the program. 

2.5. defines management system to be used. 

2.6. indicates how and by whom the program will be evaluated. 

2.7. indicates who will be responsible for monitoring students' progress. 

2.8. indicates when and how counselors will maintain their professional competencies. 

2.9. indicates the ethical guidelines for the program. 

3.0 Conceptual Model 

A conceptual model of guidance provides the program with a structure for the 
definition of goals and related competencies. This structure defines the domains 
(education, career, personal/social, leisure), the developmental level (K 12, staff, and 
parents), and the types of competencies (knowledge, attitudes, and skills). 

Criteria. The conceptual model 

3.1. focuses the program on all domains of guidance. 

3 2. provides a structure to organize goals and competencies. 

3.3. identifies the developmental focus for the guidance program. 
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4.0 Goals 

Goals, an extension of the purpose statement, define the specific desired student results 
to be achieved by the time the student leaves school. The goals are stated in terms of 
the knowledge, attitudes, and skills necessary for the student to become successful in 
the world of higher education, advanced training, the world of work, or other 
meaningful endeavors in the adult world. The goals are written in global terms for each 
of the domains identified in the conceptual model, 

Criteria. Goals 

4.1. are expressed in terms of what students are expected to achieve. 

4.2. are linked to the statement of purpose. 

4.3. include learning to learn (educational domain), work (career domain), relate 
(personal/six;ial domain), and use leisure time effectively (leisure domain). 

5.0 Competencies 

Competencies are developed proficiencies (knowledge, attitudes, and skills) that arc 
observable, transferable from a learning situation to a real-life situation, and directly 
linked to a guidance goal, Competencies arc used by counselors, students, parents, and 
staff to measure whether students arc moving toward the stated goals. Competencies are 
sometimes referred to as formative indicators or as behavioral objectives. 

Criteria. Competencies 

5.1. are organized into content areas as defined in the guidance goal. 

5.2. are ordered to match developmental stages of students by grade level. 

5.3. are measurable, 

6.0 Management System 

The management system establishes accountability for specific results and indicates 
who will be responsible for students acquiring predetermined competencies, which 
data will be generated, how the data will be collected, and when the data will be 
submitted to the administrator. There is a clear division between the student results, for 
which counselors assume accountability, and duties assigned by administration. A clear 
definition of the reciprocal responsibilities of the program manager (usually a school 
administrator) and the counselor facilitates communication and the flow of data— vital 
to making necessary mid-course corrections in program plans. 

Criteria. The management system ensures that 

6.1. each counselor has responsibility for producing specific student results. 

6.2. all goals are addressed. 

6.3. all students arc included in the agreed-upon results. 

6.4. there is a clear division between student results, for which counselors assume 
responsibilities, and duties assigned by administration. 

6.5. there is an established timeline for using and for reporting assessed evidence of 
results to appropriate administrative personnel. 

6.6. there is a systematic method of monitoring guidance activities. 

6.7. each step of the management system prixess has reciprocal responsibility by the 
counselor and program manager. 

6.8. each counselor produces a plan for how guidance results will be accomplished. 



7.0 Results Agreements 

Results agreements are responsibility statements formulated by each counselor that 
specify the results for which she/he will be accountable, Results agreements are 
expressed in terms of the competencies students will achieve, are closely linked to the 
program goals and include a separate section for all assigned duties. The administrator 
responsible for the guidance program is active in the negotiation of the results 
agreements, and the program manager audits the results agreements to ensure that the 
assigned duties do not place a disproportionate burden on teachers, 

Criteria. In drafting a results agreement, it is crucial that 

7.1. specific desired results arc delineated for students, parents, and staff. 

7.2. results are stated in measurable terms so that results can be documented. 

7.3. the agreement is consistent with and related to program goals and competencies. 

7.4. the program manager (administrator) is actively involved in the negotiations of the 
results agreements. 

7.5. the negotiated results agreement is validated by the signatures of the counselor 
and administrator. 

8.0 Needs Data 

Needs— the gaps between desired results and results being achieved— are directly 
related to the goals and student competencies defined for the program. 

Traditionally, needs assessments have been used deductively, to seek information from 
program participants regarding program design and direction based on their perceived 
needs. An inductive program, on the other hand, uses needs assessment to validate the 
program goals and indicate priorities within the program parameters. Inductive needs 
assessments allow the professional staff to use their expertise and the structure 
provided by a conceptual model to design a comprehensive program to meet the needs 
of all students, rather than limiting the program design to meeting the needs of a 
particular group. Therefore, the recommended use of needs data is to complete a 
competency assessment of students to determine the amount and type of resources and 
effort needed to assist all students in achieving the expected level of proficiency. 

(A more complete discussion of various needs assessment instruments and their uses 
can be found in Appendix A.) 

Criteria. Needs data reveal whether 

8.1. the assessed needs arc expressed in results and are linked to the competencies, 
goals, and purpose. 

8.2. the needs are student based. 

8.3. the needs identify gaps in desired results- i.e., discrepancy between "what is" and 
"what should be." 




25 



2U 



9.0 Results Plans 

The plans completed by the counselors indicate how the results will be achieved. The 
plan contains the competency, criteria for success, the activities and resources, persons 
responsible for completion of tasks, and how the evaluation will be done. 

Criteria. Results plans 

9.1. are written by each counselor. 

9.2. specify competencies, criteria for success, areas of responsibility, activities and 
their timetables, resources, means of evaluation, and data collection. 

10.0 Monitoring Students' Guidance Results 

Monitoring is the process of determining to what degree each student acquires the 
specified guidance competencies. A key element is communicating to the student and 
the parent the individual student's progress in attaining guidance-related goals. The 
process includes a responsible adult (counselor, teacher, parent, employer, etc.) who 
observes or measures competency demonstration and records verification on a form or 
student folder. 

Criteria. In the monitoring process 

10.1. counselors are accountable for all students. 

10.2. there is an established method of monitoring and verifying students' progress in 
demonstrating guidance competencies. 

10.3. there is an established means to record the progress, such as: 

individual vocational/career educational plan 
educational and career planning folder 
guidance report card 

10.4. activities are delineated by grade level. 

11.0 Advisory Council 

An advisory council is a group of persons representative of those groups affected by the 
guidance program: parents, teachers, pupil personnel services staff, administration, local 
community groups and, when appropriate, students. The council's function is to audit 
guidance program results and recommend priorities to the appropriate administrative 
body. 

Criteria. An advisory council 

1 1.1. is appointed and has a published schedule of meetings and agenda items. 

1 1.2. has an appropriate representative membership. 

11.3. chair is held by someone other than an administrator or counselor. 

1 1.4. has a purpose or plan of work that is agreed upon. 

1 1.5. makes minutes of its meetings available for review. 
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12.0 Master Calendar of Events 

A calendar of events is published to inform students, teachers, parents, and 
admin istrators of activities. The calendar makes the program and the counselors' 
contributions more visible to the school, the district, and the community. The 
publication of monthly and yearly guidance calendars allows others—principals, parent- 
teacher organizations, church groups, and family-oriented community groups— to 
include guidance activities on their lists of events. 

Criteria. The calendar 

12.1. is published. 

12.2. contains events that are identified by grade, month, and activity. 

12.3. indicates date, time, purpose, and location of all events. 

12.4. is distributed to appropriate, interested individuals and groups. 

• Glossary 

The glossary clearly defines all terms used in the description of the program. The 
glossary employs precise, clear language to ensure that team members, administrators, 
students, parents, and community members understand all aspects of the program. 

Criteria. The glossary 

a. clearly articulates the program to students, parents, staff, administrators, and the 
community. 

b. is written at a basic reading level and has been tested for clarity to facilitate a clear 
understanding of the total program. 

The results-based guidance program, a system of interrelated and interdependent 
elements, is in accord with the school district's philosophy, curriculum, and other 
programs. Figure 1 shows the interrelationship of elements in a results-based system. 

• Program Evaluation 

In a results-based guidance program three types of results are evaluated. Although 
evaluation is listed as the last element of the results-based guidance system, it is integral 
to the entire program. The data flow system included in the management system 
enables counselors and teachers to evaluate results or competencies immediately after a 
guidance activity has been completed. 

Intermediate evaluation, which occurs when the student is ready to graduate from the 
school, measures a group of competencies acquired throughout the schooling process. 
Intermediate evaluation assesses the student's level of goal attainment. 

The aim of long range evaluation is to determine the extent to which the student's 
preparation during elementary, middle, and high school has provided the competencies 
necessary to succeed in a range of post high school endeavors—educational, 
(Kcupational, social, and personal. Obviously this level of evaluation occurs after the 
students have left school and had an opportunity to try their skills in the adult world. 
Three , five , and ten year follow up studies are the most common intervals for 
collection of these evaluation data. 
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Criteria. In program evaluation 

a. competencies are assessed and reported on an ongoing basis. 

b. goal evaluation is completed and reported. 

c. a systematic follow-up data collection system is implemented and findings arc 
reported. 

d. findings are used to improve programs. 

c. data and recommendations are reported to the administration, board of education, 
and advisory council. 

Although results-based guidance programs share with the other models presented in 
this Guide-din emphasis on student results as the primary' purpose of guidance, there are 
differences between the nuxlels. 

One difference between the two models presented in this Guide is the method of 
achieving the agreed upon results. In the results based model presented in this 
chapter, the strategies to achieve results are chosen by the counselor and administrator 
at the school. Although there is an implied curriculum, it consists only of a delineation 
of competencies that students are to gain. This gives counselors and program managers 
the freedom to create strategies that match their unique expertise, interests, resources, 
and populations. That the results are to be attained, assessed, and reported is not 
negotiable; how students acquire those results is determined by the people operating 
the program. 

Another major difference between this results based guidance model and other models 
that focus on results (see Chapter 3) is the importance of ongoing and periodic data 
collection. Evaluation data is used consistently by counselors to make indicated changes 
in plans. Intermediate and long range evaluation findings are used to alter the total 
program as necessary. Collection and analysis of data is an integral part of each 
counselor's planning prcxess. 
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CHAPTER 



THREE 



Guidance Program Evaluation: 
The Program Audit 

Norman C. Gysbers, University of Missouri-Columbia 



After completing this chapter, you should be able to: 

• define when a guidance program meets the basic 
criteria of a program 

• identify basic elements of a sample guidance 
program. 

• identify and use identified standards to perform a 
program audit. 

• analyze time and task distribution within a guidance 
program. 

• differentiate and explain the difference between 
processes (means) and results. 
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Guidance program evaluation examines the readiness of a program for results 
evaluation. Before the results of a guidance program can be evaluated, it is first 
necessary to ascertain that a program exists and is in place. This chapter examines the 
readiness of a program for results evaluation by focusing on three questions: 

1. When is a program a program? 

2. What are the basic elements of a program? 

3. Are the basic elements of the program in place? 

Only after these questions are answered can the results of a program be identified. 



When Is a Guidance Program a Guidance Program? 

This question assumes that program can be defined and that guidance wa program. As 
with other programs in education, a guidance program must have content and structure 
and include the following: 

• desired results (competencies) 

• activities to assist students to achieve these results 

• professionally recognized and certified personnel 

• resources (Gysbers and Henderson, 1988, 30) 

If guidance is not understood and implemented as a program— if it is seen instead as an 
ancillary service, loosely organized around broad role and function statements— then it 
cannot be evaluated as a program. Thus, establishing the concept of guidance as a 
program is the first step in the process of program evaluation. 

The following generally accepted characteristics of a program will help answer the 
question: "When is a guidance program a guidance program?" 

1. Programs are comprehensive and developmental. They are developmental in that 
guidance activities are conducted on a continuing and planned basis. While 
immediate and crisis needs of students must be met, a major focus of developmental 
programs is on systematically providing students with experiences throughout their 
schooling to help them gain competencies to meet their personal, social, 
educational, and career needs. Guidance programs are comprehensive in that a full 
range of activities and services is provided including assessment, information, 
counseling, consultation, referral, placement, follow-up, and evaluation. 

2. Programs focus on students' competencies, not just on their deficiencies. To some a 
major focus in guidance is the problems students have and the obstacles they may 
face. Addressing deficiencies is important, but should not dominate. If it is 
emphasized in isolation, attention is often focused on what is wrong with students, 
not what is right. Obviously, problems and obstacles need to be identified and 
remedied, but they should not overshadow the existing or potential competencies of 
students. A major emphasis of a guidance program is to help students develop 
guidance relatec competencies by building on the competencies they already have. 
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3. Programs mandate collaboration, A developmental guidance program is based on 
the assumption that counselors, teachers, parents, and administrators are involved in 
supporting and carrying out the guidance program. At the same time, it is 
understood that certified counselors are central to the planning, implementation, 
and evaluation of the program. They provide direct services to individuals and work 
in consultative relationships with other members of the guidance team. 

4. Programs mandate articulation. A basic assumption of comprehensive guidance 
programs is program continuity— i.e. content and programmatic linkages across all 
grade levels K 12. This means that the guidance staff across all levels meet on a 
regular basis to exchange information and to update programs as new programs and 
student needs . re identified (Drier and Gysbcrs, 1989). 

5. Programs can be evaluated. Programs have identifiable and quantifiable structure, 
processes, activities, content, and resources. 



Basic Elements of a Guidance Program 

To be complete, comprehensive guidance programs K 12 require three basic elements: 
(1) structure, processes, and activities, (2) content, and (3) resources. 

Structure, Processes, and Activities 

A guidance program needs structure so that it can function in an organized way. The 
structure and processes below were used in the program model first described by 
Gysbcrs and Moore ( 1974, 1981 ) and later by Gysbers and Henderson ( 19H8). This 
model is used to illustrate the interrelationship of program structure, activities, and 
processes. 

Structural Components 

Definition. The program definition includes the mission statement of the guidance 
program and emphasizes the centraliry of the guidance program within the school 
district s total educational program. It delineates and emphasizes the competencies 
students will possess as a result of their involvement in the program. It summarizes the 
components of the program. 

Kationale.The rationale discusses the importance of guidance as an integral but 
identifiable part of the educational system and provides reasons why students in our 
society need to acquire the competencies that will result from their involvement in a 
comprehensive guidance program. It includes implications drawn from the student and 
community needs assessments or other clarifications of goals for the local educational 
system. 
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Assumptions. Assumptions are the principles that shape and guide the program. They 
may include statements regarding the essence of the contribution that school 
counselors working in guidance programs make to students' development, the premises 
that undergird the comprehensiveness and the balanced nature of the program, and the 
relationship between the guidance program and the other educational programs A 
statement that the program needs to be continuously planned, designed, implemented, 
and evaluated should be included as well. (In Chapter 2 these are referred to as "we 
agree" philosophy statements.) 

Program Components 

Guidance curriculum. The guidance curriculum (content and instruction) is the center 
of the developmental part of a comprehensive guidance program. It contains statements 
regarding the goals for guidance learning activities and the competencies to be 
achieved by students. The curriculum is organized by grade level; that is, a scope and 
sequence of learnings for grades K 12 is established. It is designed to serve all students 
and is often called classroom or group guidance. 

• Sample processes: 

• large group presentations such as career nights or class meetings 

• classroom presentations 

• small group presentations 

Individual planning The activities of the individual planning components are designed 
for all students and are intended to assist them in the development and implementation 
of their personal-social, educational, and career plans. These activities are designed to 
help students understand and monitor their growth and development and take action 
on their next steps educationally or vocationally. The activities in this component are 
generally delivered individually, but sometimes small groups are used. 

• Sample processes: 

• advisement 

• assessment 

• placement 

Responsive services. The purpose of this component is to prov ide special help to 
students who are facing problems that interfere with their healthy personal social, 
career, or educational development. It includes the provision of preventive responses to 
students who arc on the brink of choosing an unhealthy or inappropriate solution to 
their problems or who are unable to cope with a situation. Remedial interventions also 
are prov ided for students who have already made unwise choices or have not coped 
well with problem situations. 

• Sample pnxresscs: 

• individual counseling 

• small group counseling 

• consultation 

• referrals 



9 

ERIC 



33 



36 



System support. This component has two parts: (1) activities necessary to support the 
other three components and (2) activities implemented by guidance staff that support 
other educational programs. Support that the guidance staff provides to other programs 
includes (a) the system related aspects of the individual planning activities (e.g., 
student course selection), (b) linkage with other instructional areas such as special 
education, vocational education, English, social studies, and mathematics programs, and 
(c) guidance related administrative assignments. 
• Sample pnxresses: 

• program management 

• consultation 

• community outreach 

• public relations 

• professional development 

Content 

Traditionally, guidance has been conceptualized and implemented as a support serv ice 
lacking a content base of its own (Aubrey, 19 7 3). Springhall ( 1$F1) made this point 
when he stated that guidance textbix)ks usually avoid discussion of a subject-matter 
base for guidance programs. Today, however, it is common to talk about 
comprehensive guidance programs as an integral part of education with their own 
content. 

One major feature of these programs is a fcxus on student outcomes (competencies); 
students 1 achievement of these outcomes is the responsibility of the counselors and 
other student support personnel. Knowledge and skills to be learned by individuals as a 
result of the program are variously grouped by different writers as (a ) personal, scx'ial, 
career, and educational, (b) knowledge of self and others, career planning and 
exploration, and educational and vixational development, or (c) learning, 
personal/social, and career/ vixational. These categories identify domain or content 
areas of human growth and development from which student competencies are drawn. 
Guidance activities and resources designed to assist students to achieve these 
competencies are organized accordingly. 

Resources 

Early in this chapter it was noted that guidance is a program like other programs in the 
school. Like other programs, a guidance program must have resources to operate. 
Gysbers and Henderson (1988) categorized these guidance program resources as 
human, financial, and political. 

Human Resources 

Human resources, the personnel who are involved in the program, include professional 
counselors, teachers, administrators, parents, students, and members of the community. 
Professional school counselors are central to the program and provide a variety of 
services directly to students. They collaborate with teachers and consult with teachers, 
parents, and administrators. They are accountable for the program's operation and 
results. They conduct program evaluations and evaluate the results of the program. 
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Financial Resources 

Financial resources include the following: 

1. Budget— money for personnel, resources, professional development, and the like 

2. Material— educational, occupational, and personal-social information (all types of 
media) used to deliver guidance results 

3. Equipment— machines, computers, and other equipment used to deliver the 
guidance results 

4. Facilities— space of sufficient size, appropriately located and organized to deliver the 
programs 

Political Resources 

Political resources are defined as "the support that is rooted in disrict or building 
policies, in state and federal laws, rules, or regulations, or in the standards adopted by 
accreditation or other professional associations to which your school or district 
subscribes" (Gysbers and Henderson, 1988, 102). Of particular importance are local 
school board policies that describe and endorse the nature and structure of the 
program, provide support for the program, and provide the time to cam' out the 
program. Also important are any state regulations that establish personnel ratios and 
program standards. 



Summary 

What are the basic elements of a guidance program? This chapter has suggested there 
are three basic elements: (1) structure, prwesses, and activities, (2) content, and (3) 
resources. Figure 1 presents these elements in summary form. 

Program Means 



Program 
Structure, Processes, & Activities 

I 



I 

• Structural 
Components 

Definition 

Rationale 

Assumptions 



• Program Components 
and Sample Processes 

Guidance Curriculum 
Structured Groups 
Classroom Presentations 

Individual Planning 
Advisement 
Assessment 
Placement* Follow-up 

Responsive Services 
Individual Counseling 
Small Group Counseling 
Consultation 
Referral 

System Support 
Management 
Consultation 
Community Outreach 
Public Relations 



I 

Program 
Content 



• Life Career Development 
Model Domains 

Self Knowledge * 
Interpersonal Skills 

Life Roles, Settings. 
& Events 

Life Career Planning 



Program 
Resources 



• Resources 



Human 
Counselors 
Teachers 
Administrators 
Parents 
Students 
Community 

Personnel 
Business & 

Labor Personnel 

Financial 
Budget 
Materials 
Equipment 
Facilities 

Political 
Policies 
Rules 
Laws 



Figure 3.1 
Basic Elements ol a Sample 
Guidance Program 
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Are the Basic Elements of the Program in Place? 



Once decisions have been made about the basic elements of a program, the next step is 
a program audit to establish the degree to which these elements are in place and 
operating. The three elements described previously, when used as program standards, 
illustrate how this step is implemented. In addition, the appropriate use of professional 
counselors 1 time is crucial to the successful implementation and operation of a 
guidance program. 

Pr ogram Audit Using Standards 

The use of standards is one way to see the degree to which the program elements are in 
place and operating. For the purposes of this chapter, the term "standards" is used to 
mean the basic elements and subelcments of the program. To illustrate the use of this 
procedure, the three basic elements— structure and prwesses, content, and resources- 
are defined as the standards of a guidance program, Once these example standards are 
selected, the degree to which they are in place and operating can then be measured 
using a five-point scale: l=not implemented, 2=weakly implemented, 3=moderately 
implemented, 4=strongly implemented, and S=fully implemented When a standard 
does not apply, "NA" (not applicable) is used. 

In Figures 3.2 and 33 on the following pages are some selected example standards 
grouped under two of the three basic elements and subelements of the program. For an 
actual audit, all of the elements and subelements would need to be fully addressed. The 
number of standards used in the program audit will vary depending upon the nature, 
scope, and number of elements or subelements in the program. The selected standards 
listed under the four program components are from Northside Independent School 
District, San Antonio, Texas (Gysbers and Henderson, 1988, 265-267). 

Time and Task Analysis 

A major part of any guidance program audit involves answering two questions: "How 
much time are counselors spending carrying out program tasks?" and "How is that time 
distributed?" If counselors are not spending their time carrying out the agreed upon 
program and their time is not distributed across the program components, the program 
is not functioning appropriately. In some districts, counselors may be spending from 20 
to 40 percent of their time on activities that are not part of the program. If this is the 
case, the program is not in full operation because the key persons in the program— the 
counselors— are not always available. Also, in other districts, all of the counselors may 
be spending an inordinate amount of time on one part of the program (or with one 
element of the population) at the expense of the other part of the program. This may 
not be an effectiv e use of time. 
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Structural Components 
Definition 

1. There is a written definition for the guidance program. 

2. There is a list of assumptions. 

Program Components 
Guidance Curriculum 

1. All students are assisted in a systematic way to develop knowledge, 
understanding, and skills identified as necessary to enhance their personal, 
social, career, and educational development 

2. Developmental^ appropriate student outcomes are specified for each grade 
level. 

3. Outcome selection is based on an assessment of student needs for guidance 
instruction. 

4. The guidance curriculum is taught to class-sized groups of students. 



NA 1 2 3 4 5 
NA 1 2 3 4 5 



ERIC 



3 7 

40 



NA 1 2 3 4 5 

NA 1 2 3 4 5 

NA 1 2 3 4 5 

NA 1 2 3 4 5 



Individual Planning 

1. All students are provided information and are assisted in applying the skills 
necessary to make plans and to take the next appropriate steps toward their 
established goals. 

2. The priority for the component, as indicated by the time spent by most 
counselor(s) and the activities provided, is to assist students to complete 
successfully then elementary/middle/high school education. 

3. Individual planning activities are preceded by pertinent guidance instruction. 
Responsive Services 

1. Students in need are assisted in solving immediate problems that interfere with 

their healthy personal, social, career, and/or educational development. NA 1 

2. A balance of service is maintained for students with preventive and remedial 

level needs. NA 1 

3. There is a systematic provision of responsive services as follows: 

a. group counseling 

b. individual counseling 

c. staff consultation 

d. parent consultation 
3. referral to other specialists 

System Support 

1. Communication mechanisms are established which facilitate collaboration 
between the guidance department and administration. 

2. Communication mechanisms are established which facilitate collaboration 
between the guidance department and the instructional departments. 

3. Counselor(s) is/are encouraged to utilize professional growth opportunities. 



Figure 3.2 
Example Standards tor the 
Element Program Structure and Processes 



NA 1 2 3 4 5 

NA 1 2 3 4 5 
NA 1 2 3 4 5 



NA 1 2 3 4 5 

NA 1 2 3 4 5 

NA 1 2 3 4 5 

NA 1 2 3 4 5 

NA 1 2 3 4 5 



NA 1 2 3 4 5 

NA 1 2 3 4 5 
NA 1 2 3 4 5 



Financial Resources 
Budget 

1. A reasonable budget is appropriated to the guidance department by 

administration. NA 1 2 3 4 5 

Materials and Equipment 

1. The equipment available for conducting individual planning activities is 

adequate. NA 1 2 3 4 5 

2. The materials available to support implementation of the individual planning 

activities are effective. NA 1 2 3 4 5 

3. The equipment available for conducting responsive services is adequate. NA 1 2 3 4 5 

4. The materials available to support implementation of the responsive service 

activities are effective. NA 1 2 3 4 5 

Figure 3.3 

Example Standards for the 
Element Program Resources 



As one example of how to check counselors' use of time, the Missouri Comprehensive 
' rogram Model (Starr and Gysbers, 1988) recommends that counselors complete the 
Time and Task Analysis Form (Starr and Gysbers, 1988, 93; see Figure 3.4). The form is 
used to survey the amount and distribution of counselors' time prior to the 
implementation of a comprehensive program. Because of the need to provide a 
baseline from which to plan the desired comprehensive program, the survey is done 
prior to implementation. Note that the form uses few program components— guidance 
curriculum, individual planning, responsive services, and system support. Note also that 
the inclusion of the last column— nonguidance activities— assumes that many 
counselors currently are spending time doing activities that are not part of the guidance 
program. It is important that these tasks also be identified and the time it takes to do 
them calculated. 

It is recommended that the form be filled out one day of each week for the full school 
year. Monday might be selected as form completion day for the first week of school, 
Tuesday the next week, Wednesday the next, and so on. Percentages of time would be 
calculated for the full school year for each category, and the resulting data would be 
presented for all counselors in a building or for all counselors in the district at all levels. 
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Categories 


Curriculum 


Individual 
Planning 


riesponslve 
Services 


System 
Support 


Nonguldance 
Activities 


Examples of 
Activities 


Classroom 
Activities, 
Group Activities 


Individual 

Advisement, 

Individual 

Assessment, 

Placement, 

Vocational/ 

Occupational 

Exploration 


Consultation, 

Personal 

Counseling, 

Small Groups, 

Individual 

Counseling, 

Crisis 

Counseling, 
Referral 


Research, 
Staff/ 

Community 

Development, 

Professional 

Development 

Committee/ 

Advisory 

Boards, 

Program 

Management/ 

Operation 


Bus Duty, 
Lunchroom Duty, 
Playground Duty, 
Balancing Class 
Sizes, 

Building Master 

Schedule, 

Substitute 

Teaching, 

Figuring GPA's, 

Figuring Class 

Rank 


7:00- 7:30 












7:30- 8:00 












8:00- 8:30 












8:30- 9:00 












9:00- 9:30 












9:30-10:00 












10:00-10:30 












10:30-11:00 












11:00-11:30 












11:30-12:00 












12:00-12:30 












12:30- 1:00 













1:00- 1:30 












1:30- 2:00 












2:00- 2:30 













Figure 3.4 
Time and Task Analysis Form 
30 Minute Intervals 
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Using the sample high school form from the Missouri Model, Figure 3.*> (Starr and 
Gysbers, 1988, 103) displays data from the analysis. Note the circle on the left should 
display the data from the Time and Task Analysis Form for the current program. In the 
center, recommended percentages are presented. Note also that the percentage of time 
suggested for nonguidance activities is zero. This means that when the full program 
goes into effect (see the circle on the right in Figure 3.^)* all of the counselor's time 
would be devoted to the guidance program. When this happens, it becomes a one 
hundred percent program. 





System 
Support 
15% -20% 




Current District 
Program Percentages 

guidance curriculum 

individual planning 

responsive service 

system support 

nonguidance 



Suggested 
Percentages 

15% - 25% guidance curriculum 
25% - 35% individual planning 
25% - 35% responsive services 
15% - 20% system support 
0% nonguidance 



Desired District 
Program Percentages 

guidance curriculum 

individual planning 

responsive service 

system support 



Figure 3.S 
Time Distribution Form 
High School Level 



In the circle on the right a school district would decide on and then display the 
percentages of the time it feeis should be alienated to the four program components, 
based on district needs and goals. Each building in the district would then determine 
the percentage of time needed for that building. Buildings throughout the district could 
vary in percentages of time, as could the percentages of time for individual counselors 
in a particular building. Figures 3.6 and 3.7 display similar forms for middle 
school/ junior high and elementary leveL 
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Current District 
Program Percentages 

guidance curriculum 

individual planning 

responsive service 

system support 

nonguidance 



, Individual 

Planning /SystemN 
15% -25%/ Support 1 
10%- 15% 



Responsive 
Services 
30% - 40% 



Curriculum 
25% - 35% 




Suggested 
Percentages 

25% - 35% guidance curriculum 
15% - 25% individual planning 
30% - 40% responsive services 
10% 15% system support 
0% nonguidance 



Desired District 
Program Percentages 

guidance curriculum 

individual planning 

responsive service 

system support 



Figure 3.6 
Time Distribution Form 
Middle School/ Junior High Level 




Individual 
Planning 

w5%- 10%/Syster 

Support 

[Responsive\ / 10% * 15% 
, Services 
1 30% -40% _ 

Curriculum 
35% - 45% 




Current District 
Program Percentages 

guidance curriculum 

individual planning 

responsive service 

system support 

nonguidance 



Suggested 
Percentages 

35% - 45% guidance curriculum 
5% - 10% individual planning 
30% - 40% responsive services 
10% - 15% system support 
0% nonguidance 



Desired District 
Program Percentages 

guidance curriculum 

individual planning 

responsive service 

system support 



Figure 3,7 
Time Distribution Form 
Elementary Level 
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The Time and Task Analysis Form is filled out only once in the entire implementation 
process. The reason for this is that once the program is up and running, the calendars 
counselors maintain are used to calculate their time. In the Missouri Model, it is 
recommended that counselors maintain two week calendars. At the end of each two 
week period, counselors review their calendars, assign the activities to one of the four 
program components, and then calculate the percentages of time they spent in each. 



A Final Note 

The structure of a guidance program helps to systematically organize the guidance and 
counseling processes used to achieve the results of the program. Content identifies the 
student competencies to be attained in the program. Resources specify the necessary 
personnel, money, material, equipment, facilities, policies, and support required to 
carry out the program. These elements represent the requisite means of a program, 
without which it is impossible to fully evaluate the results of a program. 
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CHAPTER 



FOUR 



Assessing Results 

C. D. Johnson, Johnson and Associates 



After completing this chapter, you should be able to: 

• give examples of the connections between a 
program's purpose or mission and the purpose or 
mission of the school district. 

• define four kinds of results and describe how to 
establish criteria for each. 

• identify three ways to collect data. 

• give examples of a total results plan that includes a 
sample competency, what is to be learned by the 
student, who is accountable for ensuring that the 
competency is achieved, when it will be 
accomplished, what criteria will be used to 
demonstrate successful completion, and how the data 
will be collected. 

• list examples of program goals and the criteria used to 
determine success. 

• identify potential unplanned-for results (impact results) 
and how the data can be collected. 

• identify evaluation strategies for long-range results. 

• differentiate between the kinds of results for which 
counselors are responsible and unplanned-for results for 
which counselors are not accountable. 
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In this chapter the assessment of three kinds or levels of student results is discussed: 
immediate results of a specific process, terminal (intermediate) results when a student 
graduates from the system, and long-range results, three to five years after leaving the 
system. Also included is how to determine the means of evaluation and the criteria for 
success. In addition, examples of unplanned-tor results are provided along with ways to 
determine the impact of the program on parents and staff. An explanation and example 
of how to demonstrate connections between the purpose statements of the district 
board of education and the guidance program are included. 

In Chapters 2 and 3, guidelines were provided to determine if a program is results 
based. Each of the models provided specific elements, the definitions of each, and 
criteria to use in auditing a program to determine if it is ready for evaluation. Results- 
based models have certain aspects in common: (1) each has specified student 
outcomes, competencies, and/or goals, (2) each has specified means and allocated 
resources to achieve the results, and (3) each indicates connection between the 
program elements as well as between the statements of purpose, philosophy, and goals 
of the board of education and of the student support or guidance program. 



Linkages 

All programs within a school district are legitimized through the formal statements of 
purpose and goals developed by the board of education. It is the intentional ity of these 
statements that provides a framework within which all school system elements of 
curricula, management, pupil support programs, and specific services operate. Because 
the guidance program is one element of the educational system, it must show 
connections to the district's board adopied purpose and beliefs and to each school's 
purpose, mission, and beliefs. 

An example is provided below to show direct connections or linkages that help in 
deciding w hat evaluation data are to be collected and reported, and are important in 
budget considerations. 

Statement of Purpose 

The Board of Education of AAA School District is committed to having all students 
acquire the education needed to become autonomous and contributing citizens in the 
democratic socictv in which we live. 

♦ i I 

Level 3 Results: Purpose/Long range Impact 

The guidance program of AAA School District is committed to having all students 
acquire competencies in learning, working, and relating to others in order to be 
productive and contributing members of our democratic society. 

I i ♦ 
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Level 2 Results: Goals 



All students will acquire and demonstrate competencies (knowledge, attitudes, and 
skills) in studying, taking standardized and teacher-made tests, educational planning, 
and using available resources. (This is one goal within a total program. Most programs 
will have three or four goals, e.g., educational, career, personal/social.) 



• knowledge of her/his learning style 

• skills in taking lecture notes 

• etc. . . . 

(These are sample competencies that could be included within the educational goals 
cited above. Each goal is defined by knowledge, attitude, and skill competencies for 
each grade level.) 



Results are "consequences of applying resources and pnxesses" (Kaufman, 1988). The 
organizational elements model (Kaufman, 1988) provides a paradigm that can be used 
as an organizer for terminology associated with evaluation (see Chapter 1, Figure 1.1 ). 
There are four kinds of results: (1) long-range results defined in the program purpose 
and mission statements, (2) goals or intermediate/terminal results, (3) competency or 
immediate results, and (4) the unplanncd-for results caused by any of the previous 
three. 

1. Long-range results. The longrange results, as defined in the purpose and mission 
statements, refer to the intentional ity of the program. These broad statements specify 
how the program should affect students* actions one, three, five, and ten years after 
graduation. Long range results occur as the consequence of all of the student's 
educational experiences both within the system and after leaving the system. The 
purpose of evaluating the long-range results is to determine if the guidance related 
competencies have been used by the student to reach the desired intention of the 
program. 

2. Goals or intermediate results. The content of the goals or intermediate student 
results specifies in general terms what all students are expected to do because of the 
guidance program upon graduation from school. For example, upon completion of 
high school, students have demonstrated skills in using self-characteristics, 
career/occupational information, and educational skills in selecting a career or 
educational program leading to a career. Other examples include all students having 
skills in preparing resumes, interviewing for work, and knowing where to kxrate 
educational opportunities. 




Level 1 Results: Competency/Outcome 

Each student in grade will demonstrate: 



Assessing Results 
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3. Competencies, outcomes, immediate results. These are the knowledge, skills, 
and attitudes students acquire as a result of participating in a guidance activity. These 
results ire assessed during and/or immediately upon completion of an activity. For 
example, when the student completes a group guidance activity on how to prepare 
for a test, the counselor immediately assesses the student's knowledge and skills in 
that area. 

4. Impact or unplanned-for results. Unplanned-for results or impact results are 
those that occur as a consequence of a planned, implemented, and evaluated results- 
based program. For example, if all students in a given grade have demonstrated 
skills in taking standardized tests, mean test scores may rise. When all students 
acquire skills in solving personal problems, there may be fewer referrals. When 
students acquire skills in educational planning, there maybe a reduction in failures. 
When teachers acquire different classroom management skills, there may be fewer 
discipline referrals to the administration. As a result of a peer helpers program in 
grades K-12, the schools might have fewer dropouts. 

It is important for counselors and administrators to agree, however, that counselors are 
accountable only for predetermined immediate and intermediate results, not for 
unplanned-for results. For example, counselors can be accountable for all students 
demonstrating study skills that match their assessed learning styles, but they will not be 
held responsible for producing a rise in students' grades, achievement scores, or 
homework completed because other variables intervene, such as classroom teachers' 
inputs, home conditions, peer pressures, etc. Counselors can be held accountable for all 
students having skills in educational planning, but they are not accountable for students 
failing to meet classroom achievement standards. 

The who, when, what, and how of evaluation is presented in Chapter 1. It is important 
that the counselor and administrator of guidance or student support programs agree on 
who is accountable for which results. 



Establishing Criteria 

Criteria are the evidence used to determine that the students acquire the designated 
knowledge, skills, and attitudes. Although the criteria for different kinds of results 
(immediate, intermediate, and long range) may vary, ril must show linkages with the 
others. That is, the criteria of immediate results should link with the intermediate or 
goal results, which in turn link with the desired long-range behaviors, such as becoming 
self-supporting, becoming a lifelong learner, adapting to scxietal changes, and being an 
active citizen by voting and contributing to the community. 

For example, the consequence of identifying options for post-high school endeavors 
(an immediate result or competency) may lead the student (along with a number of 
other immediate results) to develop a post high school plan and take the first steps of 
application (intermediate result or goal). The criterion is that the student be able to 
demonstrate planning and application skills. The long-range result would be that the 
student enter and complete advanced training and use the skills to become a 
competent, contributing adult in his/her field of choice. 
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There arc three ways to collect data: ( 1 ) students write something, (2) students talk 
about something, and (3) students demonstrate behaviors that can be observed The 
means of collecting information are frequently determined by the pnxess of activity 
used to teach the competency 

1. Students write. Students can be asked to take a test, complete a worksheet, or 
simply list specific content requested. The written assessment can be as simple as 
one or two items, such a? (1) listing the five steps in making a career decision or (2) 
identifying six sources of assistance available for a student with personal problems. 
Answers can be geared to machine scoring, or may be teacher validated through a 
classroom quiz. 

2. Students talk about something. Students might be requested in classroom 
discussions, in small groups, or in individual conferences to describe or discuss 
specific content. For example, teachers, counselors, or peer helpers might request 
students to describe the steps to apply for work, to apply for financial assistance, to 
access help for personal problems, or to describe how they study based on their 
(assessed) learning style. Their answers validate their knowledge acquisition. 

3. Observe student behaviors. Students can be observed using skills or knowledge 
or demonstrating attitudes that have resulted from a guidance program activity. 
These observations can be made by counselors, teachers, parents, or any other 
responsible adult. Examples include observing students using resources in the 
career center, joining new social groups, being more active in classroom discussions, 
being friendly with others, or becoming involved in community activities. 

The three means of validating students* acquisition of predetermined guidance 
competencies or outcomes requires planning by counselors, as well as a trusting 
participatory relationship with school administrators. 




Puffing If All Togefher 



Examples of how to validate the three levels of results for different populations are 
presented below. Each example includes the desired results, what is to be learned, who 
is accountable, when it will be completed, the established criteria, and how the desired 
behavior will be validated. 

Immediate Results/Competencies 





Competency 


What 


Who 


When 


Criteria 


How 


A 


Ninth grade students 
will demonstrate skills 
in using sources of 
career information. 


Uses a minimum of four 
resources to locate the 
following elements for a 
career of choice: 
education 

requirements, physical 
requirements, local 
availability, pay, 
advancement 
possibilities 


Counselor 


End of unit in career 
planning in a ninth 
grade English class 


• Uses four resources 

• Identifies at least one 
career choice 

• Includes four of the 
following five from 

vuvl 1 1 vOvUI 

educational 

requirements. 

physical 

requirements, local 
availability, pay, 
advancement 
possibilities 


Student completes a 
career research paper 
listing criteria and 
references and is 
vacated as correct by 
the career center 
technician or 
counselor. 


B 


Eighth grade students 
will demonstrate skills 
in taking standardized 
tests. 


• Follows instructions 

• Paces 


Counselor 


tno or practice unit 
and during test taking 


• Homnlptpc* 

information section 
correctly 

• Begins and ends on 
time 

• Makes erasures 
correctly 

• Completes all 
sections on practice 
test 


Counselor and teacher 
observe students 
during sample testing 
session and during 
testing. 


C 


Ninth grade students 
will demonstrate skills 
in listening to others. 


Reflects feelings and 
ciarifies content 


Counselor 


During unit on 
listening when using 
videotape 


• Correctly reflects 
feelings at a level 3 
or better on the 
Carkhuff Scale 

• Reflects content and 
provides feedback a 
minimum of four 
times 


Counselor validates by 
observing students 
during activities and by 
viewing students on 
videotape. 


D 


Third grade students 
will identify how others 
contribute to them. 


• Parents take care of 
them. 

• Teachers help them 
learn. 

• Peers play with them. 

• Policemen protect 
them. 

• Musicians provide 
entertainment 


Counselor 


At the end of an 
activity on "Belonging" 


• List correctly at least 
one contribution of 
each of five persons 
contributing to the 
student according to 
the master list in 
"Belonging" materials 


Counselor will validate 
that student correctly 
completed the 
worksheet. 



Potential impact or unplanned-for results as a consequence of A-D above: 

A. Fewer program changes in grade 10. Increased enrollment in vocational courses. Decrease in 
failures. 

B. Improvement in test scores. Fewer errors on test directions. Fewer students absent on testing day. 
C Fewer referrals for interpersonal conflict with other students and/or teachers. Better or improved 

school climate. Fewer suicide attempts. More self-referrals to guidance. Fewer students being alone 
during the school day. 

D. Fewer conflicts between students. More/better self-esteem. Increase in parent involvement. 




Intermediate Results or Goal Evaluation 



Intermediate results are usually evaluated during the students' senior year, the last year 
in a specific program, or at the end of a project. These data will be collected from 
twelfth graders, eighth graders for middle school, or fifth graders for elementary school, 
students completing two years at a community college, students finishing a trade or 
technical course, or students participating in a special program. 

The following provides examples of what is evaluated, the criteria of success, who will 
collect the data, when it will be collected, and some ideas on how it can be collected. 
Data on all goals can be collected at the same time, or separate strategies can be used 
for specific needs. 

A. Educational Goal. Example: 

All students will acquire knowledge, attitudes, and skills in studying, taking 
standardized and teacher* made tests, using resources, and developing an individual 
educational plan. 

Criteria 

Study skills: Knows own learning style in terms of physical environment, time of day 
preference, organizational preference, with or without peer or adult present. Has a study 
plan based on learning style, including preparation, organization, place, time, etc. 

Test taking skills: Follows directions, paces, controls stress, prepares (sleep and 
nutrition). 

Resources: Demonstrates knowledge and use of guidance materials such as the 
Occupational Outlook Handbook, computerized college and career data bank, college 
catalogs, and financial aid information. Can identify educational resources used during 
high school career and resources available after leaving high schcxil. 

Planning: Determines own goals, identifies steps to reach goals, evaluates current and 
potential progress, identifies constraints, makes corrections as required. 

B. Career goal. Example: 

All students will acquire knowledge, attitudes, and skills in planning and preparing for a 
career that relates to their career goals and assessed aptitudes and interests. 

Criteria 

Self-characteristics: Knows own academic, work, and interpersonal aptitudes, interests, 
environmental preferences, work styles, and work values. 

World of work: Knows and uses career information sources. Identifies careers that 
match his/her self-characteristics and is aware of projected future for the specific career 
fields of interest. 
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Career decision making and planning: Has used self characteristics data and world-of 
work information to select potential career fields, knows educational and training 
requirements to enter identified careers, and has completed a written plan that outlines 
decisions and consequences necessary to achieve the career goals. 

Finding, keeping, changing employment: Has completed a current resume that includes 
work skills, work chronology, career goals, and references. Demonstrates appropriate 
job interview skills. Knows on-the-job work skills necessary to maintain employment 
and those necessary to advance; knows appropriate etiquette in leaving employment. 

Personal/social goal. Example: 

All students will acquire knowledge, attitudes, and skills in interpersonal relationships 
and in recognizing the contributions of self and others. 

Criteria 

Interpersonal relationships: Demonstrates listening skills; knows own communication 
style, strengths, and weaknesses; knows resources available for help with personal 
problems; demonstrates conflict resolution skills through listening, giving feedback, 
resolution of differences. 

Recognizing contributions: Demonstrates a sense of belonging to more than one group* 
identifies own contributions with others and others contributions with him/herself; 
knows what the different religious, racial and cultural groups contribute to scxMety; and 
demonstrates knowledge of the importance of having and being a mentor. 

Potential impact or unplanned-for results as a consequence of A-C above: 

A. Learning to learn (educational goal): Increase in graduation rate. Higher percentage 
of students planning to attend college or other post-high school educational 
program. Increase in test scores. Increase in schcx)l attendance. Increase in parent 
participation. Increase in positive school climate indicators. 

B. Learning to work (career goal): Increase in enrollment in careeroriented/vwational 
courses. Decrease in program adjustments or changes. Increase in student 
employment. 

C. Learning to relate (personal/social goal): Increase of student involvement in school 
related activities. Decrease in referrals for interpersonal conflicts. Increase in self and 
other referrals to counselors for personal problems. Decrease in suicide attempts. 

Who: Counselors, teachers, administrators, central office personnel, parent volunteers, 
consultants, members of the Guidance Advisory Council, employers, community agency 
volunteers. 

When: During the senior year, preferably late winter or early spring. 
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How: Sample strategics: 

1. Develop and use a structured interview to collect the desired data. During the 
interview the student's resume or educational/career folder with evidence of goal 
attainment should be available; e.g., student resume, college acceptance letter, etc. (If 
the number of students is large, use of a sampling method is recommended.) 

2. The parents/guardians of students interviewed can be contacted by phone, in person, 
or by mail and asked the same questions the students were asked. This allows for 
additional validation of the results, as well as determining if the parents know what 
guidance competencies their children have acquired. 

3. Another means of collecting data is to have students complete an information form or 
survey that requests evidence or demonstration of the skills they have acquired. For 
example, employment in a summer job is evidence of job-finding skills; acceptance 
to a college of one's choice is evidence of planning and application skills; a grade 
point average commensurate with student's ability is evidence of appropriate learning 
skills. 

4. Teachers can also validate that students have demonstrated the outcomes or 
competencies during class-related activities. 

5. During the spring semester of the senior year, the evaluator(s) (counselor, teacher) 
will interview a sample of students in groups of two to six during which time 
students will demonstrate personal/social skills, e.g., listening, resolving conflicts, 
verbally identifying the contributions of self and others. 

Long-range, Purpose, Follow-through Evaluation 

Ix)ng- range evaluation is conducted to determine if the program purposes have been 
achieved. To do this, it is necessary to conduct follow-up data collection. This can be 
done on a one-, three-, five-, ten year, or other timeline configuration established by the 
person or persons in charge. The data to be collected should be linked directly to the 
content or intentions of the program's statement of purpose. In the follow-up, the kinds 
of information requested will probably vary by year, i.e., year one information will differ 
from year five information, which will differ from year ten information. 

Example 

Purpose: The guidance program is designed to ensure that all students acquire the 
insights and knowledge they require to become autonomous, mature members of the 
adult society in our democratic nation. This knowledge is enhanced by students' 
acquiring transfer skills in learning, working, relating to others, and using leisure time 
constructively. 



What: Transfer skills in learning, working, relating, and leisure to become autonomous 
adults. 

Year 1 • When you graduated from high school, you indicated you were planning to 
go to (work/school). What are you doing currently? (Working full/part 
time, school Ml/part-time, nothing, military service, married. . . ) 

• The guidance program was designed to help you acquire skills in studying 

and finding employment. Do you have skills in studying? Yes No 

If yes, give examples of when/how you have successfully used your study 
skills. If no, what would you recommend school personnel do to help 
students? 

• Do you know how to find employment? Yes No Give examples 

of how you have demonstrated this skill. 

• What additional skills would you have liked to get in school? 

How: The collection of information on graduates must be carefully planned in order to 
get adequate samples. Listed are some examples for consideration: 

1. During the senior year or close to the end of the training program, identify a 
representative group of students. Meet with them individually or in small groups and 
solicit their support for participation in a follow-up data collection effort. Explain to 
them the reason for the activity and why they were selected. Include a handout on 
sample information and what they will be expected to do. Consideration should be 
given to supplying two or more addressed postcards for them to use when and if 
their address changes. 

2. Some successful efforts of quality programs include: 

• A group of seniors conducts telephone interviews of former students. 

• A group of parent volunteers conducts telephone interviews. 

• Graduating seniors complete self-addressed envelopes. During the spring of the 
following year, mail the first year follow up questions to them, including a 
stamped and addressed return envelope. 

• Vocational education programs collect follow up information on their students. 
The forms can be used to include requests for information helpful in guidance 
program evaluation. 

• In many communities, Ux'al service organizations subsidize follow up efforts by 
providing funds for hiring persons to help with mailing, telephoning, and data 
counting. They also furnish free printing, postage, and computer use. 

• Teachers are asked to call former students. 

• Efforts might cause the first, third, and/or other years' follow up to be a classroom 
project for courses such as statistics, sociology, or psychology'. The data and the 
collection processes are used to teach specific content to students. 
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Year 3 • When you graduated from high school, you indicated that you were 

planning to What are you currently doing? 



• What are your career plans? 



• Do you know where to get assistance in finding (different) employment? 

• Name those you have used 

• Do you know how to get assistance in finding additional training for your 
current or desired career or occupation? Yes No 

• How many different jobs have you had since you completed high 
school? Please list them 



• Do you plan to change your career? Yes No If yes, to what? 



• Do you plan to move to another city? Yes No If yes, we will send 

you a postcard to mail with your new address. 

• Do you know how to prepare a resume? Yes No 

• Do you have a current resume? Ye \ No 

• Do you know where to find help for personal problems? Name some 
places 

Year 5 • Are you currently employed? Yes No Full- or part time? 

• If employed, what is your job title? . 

How many hours per week do you work? 

• If you are going to school, what is your major and when do you plan to 

graduate? Major Completion date 

If you have graduated, when? 

Degree Major 

• What are some recommendations you can make to assist the guidance 
department to help its current students? 



How: (See Year 1 above.) 

Year 10 • Are you currently employed? Yes No 

If yes, is your salary adequate to cover your current living costs? 

• What is your current job title? 

• Did you attend or are you attending a community college, four-year 

college/university, or trade/professional school? Yes No 

If yes, please give the name of the school(s) 



If yes, did you graduate? Yes No How many years did you 

attend? Are you currently attending? Do you plan to 

continue your education? Yes No 
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How: One of the ways to collect information for years 5 and 10 is during preparation 
for, during, or after a class reunion. There are always students who maintain a sense of 
belonging to a school or program and who keep track of where former graduates live 
for purposes of reunions. These individuals are a great source of assistance in data 
collection efforts. 

Who: Accountability for follow-up efforts logically rests with central office personnel. 
However, many small school districts leave such efforts to school-based individuals 
such as counselors, current students, and parent volunteers. 

Potential impact or unplanned for results as a consequence of follow-up 
efforts: 

More support for local schools including tax initiatives, involvement by former students 
in school-related activities, increase in volunteers, donations of money and resources. 



Summary 

Results based guidance programs are those that clearly define what students will learn, 
the skills they will acquire, and the attitudes they will develop. There are three kinds of 
results; (1) immediate, (2) intermediate, and (3) long-range. In addition, there are 
unplanned-for results that occur as a consequence of the guidance program, such as an 
increase in attendance, more parent involvement, fewer classroom teacher referrals, a 
rise in standardized test scores, or a more positive learning/school climate. Although it 
is important to collect and review each kind of result, it is most important that 
counselors and others understand that the guidance staff is accountable only for 
"planned for" guidance program results. 
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CHAPTER 



FIVE 



Using Evaluation Results 

Edwin A. Whitfield, Ohio Department of Education 



After completing this chapter, you should be able to: 

• identify two types of evaluation and purposes for each. 

• identify three types of data used for evaluation. 

• describe several sources and methods of measuring 
student results. 

• list three uses of evaluation data. 

• define how evaluation is used. 

• describe decisions and actions to be taken based on 
four types of evaluation results. 

• identify various audiences concerned with guidance 
evaluation findings. 

• describe the key sections needed in an evaluation 
report. 

• name strategies beyond the written report to present 
evaluation information to various audiences, 



58 



Recently, at a local school board meeting, two groups presente m information to their 
board of education. Both were requesting an increase in staff. 

The school district's guidance department presented its case, explaining that additional 
counselors would allow the guidance staff to see more students, provide additional 
activities such as career or college days, increase the testing program, do more 
individual counseling, and provide more group guidance activities. All of these 
activities, it was pointed out, were well received by students and the staff. 

The second group, composed of teachers and curriculum specialists, although not 
mentioning whether students liked the instruction they were receiving, did note a 
recent decline in district achievement test scores and the fact that students were not 
achieving in relation to their potential. Class size, it was pointed out, prevented teachers 
from spending more time with underachievers and from providing extra time with 
gifted students or students experiencing learning difficulties. A lower pupil-teacher ratio 
would help increase achievement and reduce learning difficulties. 

Both of these groups were asking for the same thing from the school board— an 
increase in staff. The counseling group based its request on the opportunity for 
counselors to do more for more students. The teachers' group based its request on an 
expected increase in pupil learning and achievement. Counselors assumed unstated 
results that would flow from their processes while the teachers stated outcomes that 
would flow from a somewhat undefined process. The board decided to reduce the 
pupil-teacher ratio and leave the counseling staff unchanged. 

The scenario above, although oversimplified, describes an actual event and points out 
the need for competency-based results and their use in seeking to improve the effects 
of guidance programs in the schools. 

Although the extent of the evaluation program in the above example is uncertain, 
evidence of both product and process evaluation seems to be presented. The 
counselors presented some evidence of satisfaction with their process but ignored any 
evidence of its effectiveness or effects. The teachers provided data that the effects of the 
present program (and ratio) were less than desirable and offered a somewhat untested 
prcxess that would result in more desirable effects. 

Both product (student results) and process evaluation are essential in any program. 
Both are necessary to prove the effects of the guidance program efforts and to improve 
the effectiveness of strategies, techniques, and resources. 

Although both process and product evaluations are desired in every program, the 
preceding example shows that without effects data, the value of a program is based on 
assumptions, and any improvement, expansion, or change will be decided more on 
hope and charity than on faith that the program has desirable, tangible effects on 
learning and student development. 

This chapter will deal with the purpose and use of both process and product evaluation 
data with a variety of audiences. It will address both use of evaluation to improve the 
current guidance program as well as to prove that the program does produce desirable 
effects for the specific target groups (students, parents, teachers) of the program. 
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Types of Evaluation 



In using the results of evaluation, both formative and summative data can be utilized. 
Formative evaluation data address the implementation pnxesscs and movement toward 
the attainment of results while the program is in progress. It is concerned with both 
what the program is doing and how it is being done. Ongoing program modifications 
are generally the result of formative evaluations. 

Summative evaluation is concerned with the program's future. It addresses the levels of 
achievement of program goals, student client result, and the success or failure of the 
program prcxess. 

Both formative and summative evaluations include product, pnxess, and context data. 

Product data fixus on the outcomes, results, or prcxlucts of the program. These data 
must be related to the program goals and objectives and are used to measure how well 
students clients achieved the expected results. 

Process data focus on the program activities and prcxedures. These data are used to 
determine the effectiv eness and suitability of the program activities. Pnxess data can be 
derived from answers to questions such as: 

• Do the program activities deliver the expected competencies? 

• Were activities conducted on schedule? 

• Did each staff member carry out his her responsibilities? 

• Was there interprogram articulation? 

• Did guidance personnel utilize staff development opportunities? 

Context or resource data frxus on the environment in which the program activities 
take place. Facilities, policies, organization, attitudes, budget, and administrative and 
staff support are examples of these data (California State Department of Education, 
1981). Answers to the following types of questions can provide context data for the 
evaluation. 

• Is there a designated, recognizable program leader? 

• Is there time allotted for leadership functions? 

• Which individuals have a role in staffing the program? 

• Do counselors have the time and expertise required? 

• What physical space is available? Is it at least adequate and private? 

• Is the equipment adequate? 

• Are materials adequate? 

• Are supplies adequate? 

• Is the budget adequate? 

• Is the program equal in status to other programs? 

• How is the program coordinated with other programs? 

• Does the program include regular staff development opportunities? 

The descr.pt ion of the types of evaluation and data serve to remind us that the uses 
made of evaluations vary with the reasons for the evaluation, the types of data collected, 
and the intended audience. Evaluation results are used for two purposes. The primary 
purpose is to improve the existing program to better achieve the desired expected 
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results of that program. A second purpose, one that is be ;>ming increasingly more 
popular as national and state committees continue to call for educational reform and 
accountability, is to prove that the guidance program does produce student client 
competencies and long-range results. Both purposes must be included in the design of 
the evaluation program and the plan for the utilization of the results of the evaluation. 

The Florida Department of Education (1989) outlined both product and priKess 
evaluation guides for its career development program. Many of these guidelines are 
applicable for all domains addressed in a guidance and counseling program. The 
Florida guidelines emphasized that the student competencies must address both the 
knowledge, attitudes, and skills of students and their application by the students. They 
point out that measuring student results is similar to measuring achievement in other 
areas of the school curriculum and that measurement can take many forms. The sources 
and methods of measuring student results could include: 

• Standardized Instruments. These are catalogued and described in numerous 
professional publications. They must, of course, match the predetermined expected 
results. 

• State Assessment Programs. Data collected in these programs may be directly related 
to the evaluation questions being asked by counselors. 

• Locally Developed Measures. The staff can design measures for specific results. 

• Interviews. Structured interviews can provide data about achievement levels and 
allow for further information. 

• Diaries and Journals. Self -reporting logs can measure student activities, attitudes, 
and knowledge. 

• Checklists. These are a convenient way to record behavior events and skills. 

• Observations. Methods for recording observations can range from systematic 
observation instruments to audio or videotaping. 

• Simulations. Simulated situations can be used to provide a stimulus for assessment 
items that measure students' understanding. 

• Role Playing Situations. These can provide an opportunity for students to 
demonstrate behaviors. 

• Planning Forms. A variety of forms can allow students to describe their plans or 
decision-making results. 

In addressing process evaluation, the Florida guidelines suggested that the following 
questions be answered by the evaluation: 

• Were the activities (program) fully implemented according to schedule? 

• Did the pr<x*ess accomplish the delivery of the product (results)? 

• Was the program structure fully implemented? 
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The Florida guidelines specified three uses of evaluation data: 

1. Monitor the achievement of individual students on competencies and develop a plan 
for increasing achievement on competencies. 

2. Determine the effectiveness of the program for enhancing student attainment of 
local student development standards. For example: 

• What were the attainment levels of students on local standards? 

• Which competencies were most improved? 

• Which were least improved? 

• Were planned activities fully implemented? 

• What are the priorities for program improvement? 

3. Communicate the strengths, benefits, and needs of the programs by using the 
program plan and evaluation results with groups such as the school board, 
administration, parents, and community to increase awareness and understanding of 
the program, to increase support, and to justify resources. 

Although the Florida guidelines emphasize change or improvement on the part of 
students, product evaluation must also specify and report the level of achievement 
reached by all students. Acceptable standards or expected levels must be the ultimate 
goal regardless of the range or percent of improvement and change. 

Chapter Six addresses several evaluation models and the elements of which they are 
composed. Although differences are evident among the various ways the elements of an 
evaluation are categorized, all the models conclude with the need to utilize the findings 
or results of the evaluation. This step is the main purpose of this chapter and the 
clement that leads to formulating a course of action regarding program activities, 
resources, and expected results. Figure 1 displays four general categories of results from 
process and product evaluations that will influence the decisions and actions 
concerning the guidance program. 

Process Evaluation Results 
Positive Negative 
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Figure 5,1 
Four Categories of Results 

I 'sing Figure 5.1 it is possible to draw the following conclusions from the evaluation 
data. 

Cell 1. Both the process and product data show positive results. That is, the program 
activities are on schedule, are highly regarded by the participants, and meet the criteria 
established for the program characteristics. Products or outcomes are meeting or 
exceeding expectations. 
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Cell 2. While product or results expectations are being met, program activities are not 
effective in providing competencies as a specific result of the program activities; e.g., 
students can identify career information, but career days are disorganized and few 
participate. 

Cell 3. The program or process is functioning according to expectations, but 
student/client results or competencies are not being exhibited at the level expected; 
e.g., counselor-led classroom units in study skills are well received, but teacher 
observations indicate that students do not use the skills in the days immediately 
following the lessons. 

Cell 4. Neither program functions nor student/client results are being achieved at the 
predetermined level of expectation; e.g., copies of the traditional senior newsletter 
containing information on college visits by representatives are found scattered on 
campus, and few students attend the sessions. 

To use the conceptualization presented in Figure 5.1, the cause-and effect relationships 
between the pnxess (program) and the product (results) first must be determined. Did 
the student/clients achieve the expected competencies and are they a result of the 
program? For example, let's assume that one of the expected outcomes is for students 
to be able to correctly assess their characteristics in relation to career or training 
requirements, and that performance indicators or criteria have been predetermined. The 
next step is to identify the parts of the program that address this expectation. These 
could be a one week group guidance curriculum, a career day, use of a computerized 
career guidance system, or some combination of these and other pnxesses. 

Unless the program activities are identified and chosen to address the expected results 
of the program, it is not feasible to make decisions or take action based on the 
combined results of the pnxess and product evaluations. Product evaluation asks if 
students arc successful, while process evaluation seeks to determine why. Once the 
causeand-eflfeet relationships are established, then a variety' of decisions and actions 
can be determined depending on which cell your results fall in. 

Cell 1. Indicates both that expected results are being achieved and that the program is 
operating as planned. 

Decisiom/actiom— Maintaining both expectation levels and program functions is a 
reasonable assumption. Increasing the expectation for level of competencies and 
reassessing the needs of the students/clients are also considerations if time and 
resources permit. Expansion and growth of both expectations (competency level) and 
program are essential to viable programs. 

Cell 2. While students have achieved the expected outcomes, the program is not 
functioning as well ; s expected. Possible reasons: (a)l\\e content has not been 
covered, (b)ihc timelines and program goals have not been achieved, (c) personnel do 
not have the skills necessary to provide the planned program. 
Decisions/actions— C\ early a change in process or personnel is indicated. A different 
strategy' (e.g., small group vs. individual counseling) or different or retrained personnel 
may be needed. Since students/clients are achieving expected competencies without 
(or in spite of) the content or management of the program, possibly a reassessment of 
student needs would more clearly specify the appropriateness of the level of 
expectation. 
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Cell 3. The expected outcomes haw not been achieved, but the program is functioning 
as expected. 

Decisions/actions— The program activities must change to more directly address the 
competencies expected of students/clients who have experienced the pnxess. Possibly, 
additional content or activities need to be added to the program. Are personnel capable 
of delivering the content needed to attain the expected outcomes? Arc expected 
outcomes unrealistic? Should they be lowered? 

Cell 4. Expected results are not being achieved, and the program is not functioning as 
planned. 

Decisions/ 'actions— What other or additional content and activities will provide 
students/clients with expected knowledge or skills? Is the problem the process or the 
personnel? Are expectations too high? 

The preceding suggestions for decisions and actions emphasize the most important 
audience for the results of the evaluation program: the counseling staff. Evaluation is 
primarily used to improve the current program and consequently to have the 
students/clients achieve the levels of competencies expected of them. Prior to any use 
of the evaluation data, the counseling staff must anlay/e the results and determine how 
they will be utilized to meet the expectation that they have for students and clients. 
Once this plan has been agreed upon, it is time to turn to other audiences for support. 



Audiences 

The purpose of the evaluation and the types and complexity of the data will determine 
who will receive a report of the results of the evaluation, what will be reported, when it 
will be presented, why it will be shared, and how it will be presented. School boards, 
for example, will be more interested in cost benefits than teachers, who may value 
information on an increase in students 1 study skills, while parents might pay more 
attention to higher achievement scores. Kach report, while having common elements, 
must also be prepared with an intended audience in mind, 

School boards and administrators are similar in the types of information that they 
require. What are the overall goals for the school districts and how do the results and 
data provide support to these overall goals? Budget and staffing considerations as well 
as cost data also interest these groups. Legislators and outside funding agencies 

have a similar need for information concerning costs and accountability data. 

Parents, PTAs, community representatives, advisory committees, and news 
media have some common concerns and requirements. Achievement levels, post-high 
school choices, social adjustment, and behavior are some areas of interest to nonschool 
groups. The content of an evaluation report must address the outcomes valued by these 
audiences. 
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Teachers have a need for information concerning results that affect achievement and 
learning by the students, as well as how the guidance and counseling program affects 
and enhances the instructional program. Evaluation results that address student needs 
and acquired competencies can also serve as the content of staff development 
workshops for teachers. Teacher participation in helping students reach and 
demonstrate guidance-related competencies is increased when teachers receive and 
understand the evaluation data as it relates to student results. 

Students are interested in how the program serves them and how it operates within 
the schix)ls. They also want to know their progress in achieving outcomes. Student 
outcomes or competencies can be presented as individual report cards or progress 
reports for students and parents or as schoolwide progress reports. 

Counselors and other pupil services colleagues can use the information to 
increase their knowledge of student behavior identify professional development needs, 
' and adjust their individual and team efforts to better serve students, staff, and parents. 

It is important to know the audiences and what types of information they need to 
answer their questions concerning the products and processes of the guidance and 
counseling program. It is also important to address the needs and questions and 
anticipate the potential questions of the interested audience. Care should be taken not 
to overwhelm the audience with too much information or details that detract from its 
primary interests, decisions, or professional development. Finally, it is imperative that 
the information be timely. Program and budgeting decisions are set in school districts 1 
schedules. Information that arrives in June is of little help to school board or legislative 
budget deliberations that take place in March and April. Student achievement data, 
student learning difficulties, and competency deficiencies are of less value after 
curriculum, instructional, and course of study decisions have been made. Staff 
development programs are set to address issues and staff competencies on an annual 
basis. Information needs to be available when these decisions and plans are made. 



Evaluation Reports: Contont, Format, and Methods 

Planning how to use and report your evaluation information must be completed prior 
to any data collection. Thinking through this final step may influence the questions to 
be asked and how data are collected. 

In its Handbook for Evaluation Program Implementation (Paddis et at, undated) the 
Northwest Regional Educational laboratory suggests that the first step in summarizing 
data is to reduce the data into a concise form without sacrificing important detail. 
Questionnaires, interviews, observations, existing records, and other data should be 
tallied, totaled, converted to percentages where appropriate, and summarized. Your 
report should 

1. define the purpose of the evaluation, 

2. customize the format to the audience (decision makers may need more detail than 
others), and 
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3. provide an appropriate technique for describing the results or making comparisons 
such as: average response, range of responses, frequency distribution, percent of 
responses, degree of relationship, and/or summaries. Comparative data require 
statistical methods, and it is important to remember to present the information so it 
is clear and easily understood by the intended audience. 

Morris and Fitz-Gibbon (1978) suggest the following outline for your report. 
Front Cover. Title of program, name(s) of evaiuator(s), name of audience, period 
covered in report, and date. 

Section I. Summary. What was evaluated, why, major findings and recommendations, 
and decisions to be made. 

Section II. Background. Origin of the program, goals, students involved, context of 
the program (materials, activities, administrative arrangements), and facility and others 
involved. 

Section III. Evaluation Study. Purpose, design, outcome measures (instruments and 
data collection methods), and implementation measures (instruments used and data 
collection procedures). 

Section IV. Results. Results of outcome measurements, results of implementation 
measurement, and informal results. 

Section V. Discussion of Results. How certain is it that the program caused the 
results? How good were the results of the program? 

Section VI. Costs and Benefits (optional). Your method for calculating costs and 
benefits, costs associated with the program (dollar and nondollar costs), and benefits 
asscxMated with the program (dollar and nondollar). 

Section VII. Conclusions and Recommendations. Conclusions, recommendations 
regarding the program, and recommendations concerning subsequent evaluations. 

Pfister (1985) recommended a similar but simpler set of criteria for communicating 
evaluation data: 

1. An introduction made up of a statement of purpose and brief program description. 

2. Recommendations that arc directly related to findings (in the author's opinion the 
most important section of the report, which must precede any details concerning the 
evaluations). 

3. An evaluation design and procedures description that includes instrumentation, data 
sources, and any limitations of the evaluation (stated as simply and concisely as 
possible). 

4. Findings that report results free from recommendations and explanations. 
Pfister also recommended that a common executive summary listing key findings and 
recommendations be presented to all audiences to minimize expense and to avoid any 
misinterpretations that might arise if not all audiences receive the same information. 

Gysbers and Henderson (1988), in considering the variance in the interests and levels 
of understanding of different audiences, have suggested that a separate report 
appropriate for each group is necessary. The diverse interests, they noted, can be 
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satisfied by preparing a technical report that includes design, statistical data, and 
evaluation conclusions and a professional report that focuses on conclusions regarding 
the effectiveness of program activities and recommendations. The basic content of these 
reports would include: 

Technical reports. Description of the program, design, results, conclusions, and 
recommendations. 

Professional report. A brief summary of findings, conclusions, and recommendations. 

In addition to the two reports, Gysbers and Henderson ( \9M) also recommend that a 
vStudent Competency Reporting System be included in the utilization of evaluation 
results. The system would complement the continuous progress reporting system now 
being implemented in many instructional areas. Such a system would be a type of 
student outcome evaluation and would be based on the student competencies or 
outcomes chosen as the goals of the school guidance and counseling program. 

Several methods of reporting are contained in the examples cited above. Most assume 
that the repon will be in written form and include suggestions for technical, 
professional, and narrative approaches to a written evaluation repon. This form of 
reporting, the most common, often is required, especially in addressing school boards, 
administrators, and funding agencies. In addition, the written report avoids unnecessary 
misinterpretation that occurs when evaluation results are released to the news media, 
community groups, and other audiences that will use the results. 

In addition to the written report, it is often helpful and more interesting to present 
information in a variety of ways geared to inform, impress, and hold the attention of the 
audience. 

Oral reports— especially when accompanied by professionally prepared overhead 
transparencies (containing comparative data, graphs, and tables and employing different 
colors, if possible)— can highlight the main findings and conclusions. Oral reports have 
the advantage of immediate feedback from the audience, clarification of audience 
questions, and the public relations that come from face to face interchanges. 
Disadvantages can stem from inadequate communication skills personality conflicts, 
and the time constraints that limit the amount of information that can be delivered. 

Presentations by students are also a potentially interesting way of maintaining audience 
interest. But it is crucial that students actually demonstrate competencies and do not 
simply offer testimonials that can appear to play on the good will of the audience. 

The use of outside personnel (experts), as well as members of the evaluation team, can 
also enhance an oral presentation. Again caution is advised that any second person 
presentations must address the evaluation program and findings and not simply serve as 
a testimony to the effort and good intentions of the guidance program staff. 
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Summary 



This chapter has described two main uses of the information that results from an 
evaluation of the products and pnxess of a school guidance and counseling program. 

The first and primary use of evaluation is the improvement of the program by analyzing 
the results of the program in relationship to the pnxess or program activities. It was 
emphasized that the outcomes or the competencies (knowledge, attitudes, and skills) 
demonstrated by students drive and determine the content, resources, and activities of 
the guidance and counseling program. These demonstrated student/client 
competencies must be preceded by predetermined criteria and standards and must be 
stated in measurable terms. 

Program or pnxess evaluation is also necessary for improvement and growth of the 
program. Program activities and resources must be related to the expected outcomes 
that students will exhibit. Management, organization, student /client opinion, and 
additional context data are used in examining the process of the program. The results of 
product evaluation, when combined with the prwess results, allow for change, 
modification, or enhancement of the program and its personnel. 

A secondary but essential use of the evaluation results requires convincing others of the 
relevance of the evaluation results and the need for resources and support to effect the 
improvement and change dictated by the evaluation findings. How evaluation findings 
are presented will differ depending on the audience and the purpose of the 
presentation. 
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After completing this chapter, you should be able to: 

• identify the elements of specific evaluation models. 

• describe the differences between process and results 
evaluation. 

• explain the questions to be addressed according to 
what aspect of the program is being evaluated, 

• list sample evaluation questions to be addressed for 
different participants, i.e., students, parents, staff. 

• explain a variety of ways to collect data. 

• develop a comprehensive plan for guidance program 
evaluation. 
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Authors seem to agree on the elements of a plan to evaluate guidance programs. 
However, most authors view the evaluation process as external evaluators, not as 
persons working within the system to be evaluated. That is, their contributions are 
directed to those persons who are charged with evaluating a program in which they are 
not participating. The following presents a system that can be used by both external and 
internal evaluators. A review of models will be presented, followed by what is to be 
evaluated, sample questions to be answered, and a suggested evaluation planning 
format with examples. 

The differences among models appear to be only in attempts to categorize elements. 
For example, one model (Faddis, Evans. Hartzell, Ruzicka, and Morimitsu, undated) has 
four categories: (1) what is to be evaluated, (2) what questions are to be answered, (3) 
how the data will be collected, and (4) how the data will be used. Another model 
(Bissell, 1979) lists seven categories: (1) planning and organization, (2) designing the 
evaluation, (3) measuring project implementation, (4) measuring student performance, 
(S) measuring family, school, and community factors, (6) analyzing and reporting 
factors, and (7) using the findings. The California State Department of Education ( 1981 ) 
proposed ten categories in California's sample evaluation plan: (1) program objectives. 
(2) evaluation design, (3) assessment instruments, (4) assessment dates and personnel, 
(S) data analysis, (6) program activities, (7) monitoring dates, (8) reporting dates, (9) 
who will receive reports, and (10) the use to be made of data. All agree that the 
completion of an evaluation of a guidance program will bring all elements together in a 
cohesive system that can be used by guidance personnel and other decision makers to 
improve the existing program. 



What Is to Be Evaluated 



The evaluation of any program consists of evaluating both the results and the means 
(processes and resources) used to achieve the results. There are three kinds of results: 
immediate, intermediate, and long range (see Chapter 4). A given evaluation does not 
necessarily assess all three kinds of results. The determination of what results are to be 
assessed depends on the information required to improve a given part of the program 
at the time of evaluation. 

Long-range Results 

The desired long range results are reflected in the statement of purpose or mission of 
the program. The evaluation of the long-range results may occur yearly, in a scheduled 
sequence such as every other or every third year, or in a sequence such as one. three, 
five, and ten years after graduation. What gets evaluated is the performance of graduates 
of the program or school. The content of the evaluation is dictated by the questions 
asked that relate to the statement of purpose, e.g., "Are the graduates leading self 
sufficient lives?" "Are the graduates of our program contributing and prcxluctive 
members in their communities?" 



ERIC 



71 

70 



Intermediate Results 

Intermediate results are determined by the goals of the program that are to be reached 
by the time a student graduates or completes a set of learning activities. Evaluation of 
goals may be accomplished in two ways. The first is through an aggregate of data 
regarding the attainment of immediate goals on a yearly basis. Counselors are working 
with students at each grade level on a continuing basis throughout the year, reporting 
their results on a predetermined basis, e.g., quarterly, monthly, etc. The manager of the 
program (administrator) receives the data and compiles it to determine the progress 
toward program results. The aggregate report provides information on whether students 
are progressing toward the goal in a systematic, timely manner. When students have 
demonstrated all (or most) of the immediate results, it can be assumed that they have 
attained the intended goals. In this type of evaluation, what gets evaluated is the 
students' attainment of the defined competencies or indicators. It may be argued that 
learning all the competencies may or may not assure that students have reached the 
ultimate goal of being able to make a successful transition from high sch(X)l to their 
planned post-high school endeavors. If the long-range evaluation indicates that students 
reach success, however, then it can be assumed that the stated competencies were 
adequate to prepare them. 

The second and more decisive way to evaluate intermediate results is to assess students 
near the end of their senior year. Written surveys, interviews, review of artifacts such as 
career folders, resumes, student records of participation in extracurricular activities, 
post-high school plans, and college acceptance can be used to determine if the students 
have reached the program goals. This type of senior year comprehensive evaluation is 
usually done on a periodic basis. Sometimes it is done two or three years in a row for 
comparison and trend analysis, and then is not done for the following two or three 
years. What gets evaluated are indicators that students have knowledge, attitudes, and 
skills needed to help them reach success in their post high school endeavors. For 
example, do students have post-high school plans that match their interests, goals, and 
aptitudes? Have students applied at appropriate colleges or training programs (schcx)l 
and programs for which they are qualified and that offer the majors or options they 
want)? Do students have updated resumes? Are students active in community projects? 

Immediate Results 

Immediate results are usually stated in terms of student competencies (or performance 
indicators) and are measured on an ongoing basis immediately after a competency is 
taught. Individual counselors can use a data flow system to report immediate results 
several times a year. The program manager then compiles the data as they are reported 
and advises counselors of any problems identified as a result of the review of the data. 
For example, although plans on how counselors will reach the intended results have 
been approved in advance, necessary changes will be made during the year based on 
variables that were not anticipated. When such changes are made, there may be areas of 
duplication with other counselors or other departments, areas that inadvertently get 
neglected, or new opportunities may become available that were unknown at the time 
plans were made. The ongoing assessment and reporting of immediate results allow 
counselors to maximize their time and resources to provide the best possible program 
for students. Therefore, immediate results are continually being evaluated and reviewed 
on a daily, weekly, monthly, and quarterly basis. Immediate results are indicators that 



students are progressing toward the goals and long-range results of the program. These 
indicators are specific knowledge, attitudes, and skills measured in each goal area, e.g., 
educational, career, personal/social, and leisure. Do students have study skills, test 
taking skills, social skills, career decision-making skills, knowledge of options within 
the world of-work? Do students know their interests, goals, aptitudes, learning styles? 



Process and Resource Evaluation 

Program improvement is achieved through the results of product and process 
evaluations used to alter a process or activity, or to add or delete resources to produce a 
greater quantity or quality of student competencies or outcomes. Processes and 
resources are only relevant as they affect the assessed student results. For example, the 
evaluation of a career center (a resource) may indicate a high incidence of use and 
satisfaction with resources and services, but if students do not acquire specific 
knowledge, skills, or attitudes, the career center use is irrelevant. If the counselors 
provide all students with information on what courses to select to meet graduation 
requirements, and if results show that many of the students are unable to correctly 
select or identify what has to be taken, the prcxess, although completed as planned, is 
ineffective. 

It is essential to keep in mind that if the primary purpose of the guidance program is to 
provide all st juents with the knowledge, attitudes, and skills required to be successful 
in life, the evaluation of processes and resources must come back to the determination 
of how each facilitates the attainment of student results. Means (processes and 
resources) evaluation determines whether a given prcxess or set of resources is 
successful in providing the predetermined student results. For example, counselors 
have traditionally been hired based on a recommended counselor-student ratio, and 
recommendations for improving guidance programs have most frequently focused on 
lowering the ratio, much as teachers have recommended lowering class size. In a 
results based program, however, it is important to identify the additional student results 
that could be attained with a lower caseload or ratio of students. 

As discussed in Chapter 5, it is imperative that a cause and effect relationship be 
established between the process and resource evaluation results and the product 
evaluation results. What gets evaluated is the effectiveness of each process in producing 
the desired student results. What gets evaluated is also the relationship between the 
resources being used for the program pnxesses or activities and the student results 
attained. The purpose of evaluation is to improve the program by changing processes, 
adding resources, or both. 
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What Questions Are to Be Answered 



The questions to be answered provide direction to the evaluation of the program areas 
and the student results that the evaluation will address. Evaluation questions anticipate 
and define the possible results or products in such a way as to allow the counseling 
staff and/or others to identify and select the methods that will be used to verify that 
students have the knowledge, attitudes, and skills that constitute the selected results. In 
addition, the evaluation questions guide the counseling staff in identifying the activities 
required to address the expected student results and to judge whether the prcKess is 
functioning in the manner and to the degree that the program plan has specified. 

Questions concerning expected student results can address long-range, intermediate, or 
immediate results or competencies. They can be classified by the content or curriculum 
domain (educational, career, personal/social) of the program and by the audience to 
which they are addressed. 

Answers to the questions are typically cumulative in nature, that is, immediate results or 
competencies build to intermediate competencies which in turn contribute to long- 
range results. Consequently, internal evaluators of programs are concerned with 
identifying and verifying immediate and intermediate results. Follow-up studies provide 
verification of the long-range outcomes that are a consequence of the application and 
utilization of the students 1 knowledge, attitudes, and skills verified through the internal 
evaluation program. 

Answers to the questions concerning the process and activities of the guidance program 
will address the current strategies, techniques, and resources in place to assist students 
to achieve the results specified in the program. Although follow-up data will provide 
information for making judgments concerning the activities of the program, questions 
addressing process will concentrate on the efficiency and completeness of the pnxess, 
resources, and personnel in place to assist students to achieve the expected results of 
the program. 

Typical questions classified by domain, type, and audience follow. They are provided to 
serve as examples in formulating the questions that will be asked in order to focus the 
direction of the evaluation. 



Examples of Evaluation Questions 

Students 

1. Immediate results. (Questions are related directly to student competencies gained 
as a result of the ongoing guidance program. These competencies are indicators that 
students are progressing toward the desired goals.) 
Kducat ional domain of resul ts 

Do students make appropriate school and course selections? 
Do students know their learning styles? 

Can students identify and acquire financial assistance for post-high school options? 
Do students know how to study related to their preferred learning style? 
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Do students understand the unique requirements of a particular school for achieving 
academic success? 

Do students have standardized and teacher-made test-taking skills? 

Do students demonstrate an understanding of the importance of having salable 

workplace skills as a temporary or permanent option to college completion? 

Do students demonstrate responsible classroom and school behaviors? 

Do students know the high school course requirements needed to gain acceptance 

to the school of their choice? 

Career domain of results 

Do students know required job coping and adjustment skills? 
Can the students select a job that meets financial, occupational, and other life goals? 
Can students demonstrate interview and credential-building skills needed to acquire 
the job of their choice? 

Do students demonstrate self-confidence and self-marketing skills useful in job 
search and in job interviews? 

Do students understand the range of job options and related occupational skills 
required for obtaining and succeeding at work? 

Are students able to demonstrate career planning and decision making skills and 

techniques that will be transferable throughout their life? 

Can students identify and use data and services required to make future decisions 

concerning job change, advancement, and other career variables? 

Do students know how to deal with discrimination by supervisors or peers and with 

unethical or illegal situations? 

Can students predict causes of job stress and demonstrate skills to ward off stress 
related elements at work? 

Do students understand the interplay and effects of work life on other life roles? 
Do students understand the benefits and characteristics of organized labor in the 
work setting and ways to take advantage of its presence? 
Did students formulate career plans prior to selection of a college and major? 

Personal/ social domain of results 

Do students demonstrate self-confidence? 

Do students have conflict resolution skills? 

Do students demonstrate listening skills by reflecting feelings and content and 
giving feedback? 

Can students identify how they contribute to their school and community? 
I mpac t of unplanned for results 
Have test scores increased? 

Are more students completing classes with passing grades? 

Are there fewer failures? Better attendance? Higher enrollment in academic classes? 

Are there fewer program change requests? 

Do test scores rise? 

Do more students stay in school? 

Are there fewer referrals for classroom or school ground discipline? 
Do more students request personal counseling? 

Is there a decrease in self-defeating behaviors such as truancy, runaways, and 
suicides? 
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2. Intermediate results (goals). The following questions sample behaviors that are 
assessed at high school graduation and are a consequence of the guidance program 
in conjunction with results gained from other programs and experiences the student 
has completed. 
E ducational domain of results 

Do students have skills in learning? In legating educational resources? In test taking? 
Have students completed a post-high school plan that matches their self- 
characteristics, e.g., interests, goals, abilities? 
C areer domain of results 

Are students obtaining jobs or entering postsecondary training in the areas for which 
they were preparing? 

Is there evidence that students who have career planning assistance attend school 
more consistently, achieve at a higher level, or make more realistic selections? 
Are students able to make transitions from education to postsecondary training and 
from training to work? 

Are students aware of declining exxupations and are they planning to enter new and 
emerging fields in high technology and/or fields experiencing skill shortages? 
Do students demonstrate the ability to transfer their career planning and job search 
skills to future midcareer changes that might be required? 

Are students proficient in obtaining and using information on financial assistance for 
postsecondary vocational education and job training? 
Personal/social domain 

Do students demonstrate knowledge and acceptance of the contributions of others? 
Do students demonstrate interpersonal relationship skills? Belonging to a group? 
Do students contribute to others in their school, their community, their families? 
Do students know the resources available in their communities and how to seek 
help for personal problems from community resources? 
Impact results 

Did students apply and gain admission to postsecondary programs? 

Were there fewer dropouts, fewer program changes, and increased achievement of 

students who received guidance assistance? 

Have community statistics for teenage substance abuse, delinquency, and violence 
decreased? 

Have the dropout rate, absenteeism, and classrcx)m conflict been reduced? 

Is an appropriate level of academic achievement being attained by students? 

Are there fewer suicide attempts? 

Are there more student requests to see counselors? 

Are there fewer reported student- student conflicts? 

Are there fewer gang related incidents on campus and in the community? 
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3. Long-range results (purpose). These questions sample behaviors that can be 
used to indicate that the student's total schooling experience prepared them for life. 
Educational domain 

Are former students achieving success in their post-high school educational 
endeavors? 

Are former students well prepared for the demands facing them as college students? 
Are former students enrolling in college with a declared major? 
Are former students completing college in an appropriate time frame, with minimal 
changes in major and/or career goals? 

Are former students continuing their education after their formal programs are 

completed? 

Career domain 

Are former students assuming responsible roles as self-supporting citizens within the 
community? 

Are former students employed in satisfying jobs? 

Do former students have a plan for advancement in their field of choice? 

Are former students able to seek and gain employment in a field of interest while 

they are completing their formal education? 

Personal/social domain 

Are former students active participants and contributors to their community? 

Do former students have satisfying lifestyles, families, and/or support groups in their 

lives? 

Are former students willing to come back to the school to share their experiences 
and successes with current high school students? 

Are former students active and successful in a variety of life roles including lifelong 
learner, worker community member, citizen, family member, friend? 

Parents 

Educational domain 

Do parents know how to establish a home learning environment that is conducive to 
the student's unique learning style in doing homework? 
Do parents know their student's learning style? 

Are parents aware of the guidance program goals, philosophy, and purpose? 

Are parents aware of their students post-high school plans and do they approve? 

Are the parents aware of the resources available to them in helping their student 

make college plans and seek financial aid? 

Are the parents aware of their student's educational progress? 

C areer domain 

Are the parents aware of the careers/cxcupat ions/ jobs that are most suited to their 
student's interests, abilities, and goals? 

Are the parents aware of resources they can use to help their student's career 
development? 

Are the parents aware of the student's career goals and do they approve? 
Personal/social domain 

Are the parents confident of their own patenting style? 

Do the parents demonstrate appropriate reinforcement strategies to encourage their 
student? 
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Do the parents understand the importance of extracurricular and other positive 
social experiences for their student and do they encourage participation? 
Are the parents aware of symptoms that could indicate that their student is 
experiencing personal, social, or mental health problems? 
Are the parents aware of resources available to help their student with problems? 
Do the parents have the information and skills required 10 monitor their student's 
educational, career development, and personal/social progress? 
Staff 

E ducational domain 

Are teachers aware of the processes used by guidance counselors to monitor the 
students' educational progress? 

Do teachers use student learning styles information in planning instructional 
strategies? 

Do teachers have the skills needed for parent teacher conferences? 

Do teachers use appropriate primary and secondary reinforcers to help students 

learn? 

Are administrators and other staff members aware of how they can be instrumental 
in assisting students in the learning process? 
Career domain 

Do teachers include information on careers related to their subject areas to help 
students see relevance in what they learn? 

Are teachers aware of students career interests and stages of career development? 
Do teachers use the career center as a resource in presenting career and college- 
related materials? 

Are teachers open to including career speakers and career units as part of their 
instructional and curriculum options? 
Personal/social domain 

Do teachers use counseling and guidance resources for students having personal 
problems? 

Are teachers aware of the resources available within the school and the community 

to assist their students and them with personal concerns? 

Do teachers encourage student participation in extracurricular activities? 

Do teachers welcome counselor consultations related to problem students? 

Board of Education 

Are school board members aware of the purpose, philosophy, and goals of the 
guidance program? 

Do school bcrard members understand the need for guidance resources and how 
they are used to reach the guidance program goals? 

Arc school board members 1 expectations of guidance results commensurate with the 
stated purpose, philosophy, and goals of the guidance department? 
Are school board members able to answer parents 1 questions and community 
</ inquiries related to the guidance program? 

Do school board members agree with and support the basic purpose of the 
guidance program? 

Do school board members demonstrate their support through allocation of funds for 
necessary guidance resources such as career centers, credit for peer counseling 
classes, adequate clerical support, and inservice training? 
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Administration 

Educational domain 

Are administrators aware of the guidance program purpose, philosophy, and goals? 
Are administrators' expectations of counselors commensurate with the basic purpose 
and philosophy of the guidance program? 

Do administrators understand and support the results based guidance program? 
Do administrators provide support for the resources required to carry out the 
guidance program? 

Do administrators use counselors' strengths through providing teacher inservice, 
evening programs, group guidance, and individual counseling? 
Career domain 

Are administrators supportive of schoolwide guidance activities such as career days? 

Do administrators support the career goals of the guidance program by encouraging 

teachers to cooperate and participate in career related guidance processes? 

Do administrators demonstrate their support by providing career resources such as 

career centers? 

Personal/social domain 

Do administrators use counselors' expertise in addressing campus concerns such as 
teen pregnancy, substance abuse, absenteeism, vandalism, and suicide? 
Do administrators understand and use the counseling resources available to them, to 
the staff, and to the students through the guidance program? 

Process and Resource Questions 

When the results have been tabulated and reported, some questions emerge that 
require consideration about the use of resources and specific processes. 

The following are examples provided as a basis for identifying further questions. 

• Is the use of computers by counselors cost effective? Could this process be done 
more efficiently by clerical staff? 

• With one secretary for four counselors, would the secretary's time be used more 
effectively by assigning him/her an office and having counselors take turns 
answering the phones and students' questions, thereby reducing the counselors' 
paperwork and making them more visible? 

• Because of state and district mandated achievement testing, it takes one 
counselor's time for nine weeks a year to coordinate, disseminate, collect, and 
forward booklets and answer sheets. Would students gain more if a teacher, 
administrator, teacher substitute, or clerk did testing, thereby releasing a 
counselor to provide additional student results? 

• There is a 400:1 student-counselor ratio at the senior high and middle school 
level and a 900:1 ratio at the elementary schcx)l level. Would there be fewer 
dropouts and failures at the senior high level if the ratios were reversed, i.e., 900:1 
at the senior high level and 400:1 at the elementary level? (Away to explore 
whether good prevention cuts down on intervention.) 

• Should counselors be assigned to students? Would the program produce better 
quantity and quality if students were allowed to choose their own counselor? 
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• The current expenditure for guidance is $80 per student. Because of budget 
constraints, it is necessary to lower that to $60 per student. What resources and 
processes will be cut, or what results are most/least important and is there a way 
we can cut the budget and get the same results? 

The above are examples; however, each can be of importance in the use of data on 
student results. Each example connotes a value and a decision that will affect the 
program. The evaluators are encouraged to think through and prepare statements in 
advance on similar questions they think may be asked. 



How Data Will Be Collected 

Data collection has been addressed in part in previous sections of this book. It is 
important to note that information required to answer some questions chosen for 
evaluation may well come from a variety of sources within the schcx)l as well as in the 
community. Because student results form the basis for the program, students will be the 
major source of information both directly through demonstration of competencies 
(knowledge, attitudes, and skills) and indirectly from parent, employer, and/or teacher 
verification that the expectations have been achieved. In the same manner, if teacher, 
parent, or administration results are specified as outcomes of the program, these groups 
become a primary source of information to determine if the expected knowledge, 
attitudes, and skills have been achieved. Process evaluation questions may also be 
addressed to these same groups to determine if the program elements are in place and 
functioning efficiently and are delivered in a timely manner. 

The methods and instruments used to collect the required information will vary 
depending on the results and the audience. Figure 6.1 (on the following page) outlines 
several data collection techniques that can elicit information directly from students, 
teachers, or parents, solicit information through a third party, or provide information 
already available in existing records, reports, and assessment results. 

Some evaluation questions can be answered by reviewing data such as grade point 
averages, individual career plans, attendance reports, achievement and ability test 
results, and other available information. Other answers, however, may have to come 
from more extensive efforts such as additional standardized tests, observations, 
specifically designed tests, interviews, and other methods that require the involvement 
of students and staff. Consequently, the evaluation questions that are chosen will 
necessitate a coordinated effort by the guidance program staff and the cooperation of 
teachers, parents, administrators, and the community. Such an effort requires a data 
collection plan to answer the identified questions in the most efficient and effective 
way. 

While evaluation questions give direction and focus to the program, verifying and 
recording the answers must not detract from the process chosen and the time required 
to achieve the results or distract the students from accomplishing the goals expected of 
them. The collection of the data must be, to whatever extent possible, an integral, 
efficient, and unobtrusive part of the total guidance program. 
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Outcomes and Evaluation Methods 



Outcomes 


Possible Evaluation Method 


Application 
Concept acquisition 
Memorization of facts 
Problem solving 
Comprehension 
Skills 


Objective test, product evaluation, 
rating scale, checklist 


Performance 


Rating scale, checklist, product 
evaluation 


Classroom behavior 


Rating scale, checklist, attendance 
record 


Interest 


Questionnaire, checklist, interest 
inventory 


Attitude 


Rating scale, questionnaire, checklist, 
objective test 


Aspiration level 


Rating scale, interview, objective test 
open-ended sentences, word 
association test 


Adjustment 


Rating scale, anecdotal report, 
interview, sociogram, records 



Figure 6.1 

Source: The New England Educational Assessment Project (1966) 
A guide to assessment and evaluation procedures. Providence, Rl 



Guidance program results based on the mission and philosophy of the school district 
can be validated by much of the data collected routinely by district assessment 
programs, administrative reports, central office or building records, and teacher 
observations and reports. Using these existing data and/or including guidance program 
data needs in the existing processes can provide information with a minimum of 
change in district or school prcxedures and record keeping. 

Where additional data collection is required to verify that students have achieved 
intermediate results, consistent, systematic verification and recording of immediate 
results as they are achieved by students can lessen or eliminate the need for special 
end of year assessment, because intermediate competencies are an accumulation ot 
immediate results. Answers to questions concerning long-range results specific to the 
guidance program can be a part of the school district's follow-up program as well as a 
part of periodic community surveys. 

The data necessary to answer some questions might have to be collected through a 
separate administration of assessment instruments. Because all expected immediate and 
intermediate results are stated in measurable terms, it may be preferable to develop 
criterion referenced tests that verify or measure the achievement of students on the 
expected competencies. 
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Some existing standardized tests may also provide the data needed to measure the 
achievement of these competencies. A variety of interest, aptitude, achievement, values, 
personality, and other instruments exists that may meet the needs of the evaluation 
plan. By examining current reviews of these instruments and obtaining sample copies, 
the exact fit of these commercial instruments to your data needs can be ascertained. 

Some excellent sources of information on standardized tests are: 

A Counselor's Guide to Career Assessment Instruments, Jerome T. Kapes and Marjorie 

Moran Mastie, Editors. Published by the National Career Development Association, this 

extensive reference reviews 43 of the most commonly used aptitude, interest, values, 

career development, and personality instruments. In addition 126 other instruments are 

briefly described. Addresses of all test publishers are included. It is available from the 

American Association for Counseling and Development (AACD). 

The Mental Measurement Yearbook provides a detailed review of most available 

assessment instruments. This classic reference is updated frequently and is available at 

most libraries. 

The Association for Measurement and Evaluation in Counseling and Development 
(AMECD) provides current reviews of tests in each issue of its journal, Measurement 
and Evaluation in Counseling and Development. AMECD is a division of the American 
Association for Counseling and Development. The journal is available in most college 
and university libraries. AACD and AMECD have also published a position statement 
concerning the responsibilities of test users. They identified four major uses of tests: ( 1) 
for placement and selection, (2) to predict performance or behavior, (3) for diagnosis 
or description of behavior, and (4) to provide data to give systematic information and 
depict growth. These guidelines fexus on four general categories of tests: aptitude and 
ability, achievement, interest attitude and values, and perceptual-motor tests. 
The American Psychological Association publishes Standards for Educational and 
Psychological Testing These standards provide criteria for the evaluation of tests, testing 
practices, and the effects of test use. 

It is evident that if the evaluation plan is to be manageable and unobtrusive it must be 
an ongoing component of the total guidance program. The results expected and the 
prtxess selected are primary to what is to be evaluated and how it is to be verified. 
Evaluation questions are asked at the initiation of program planning and provide 
direction to the continuing focus of the program and the redirection of future emphasis 
of the program. 

Careful attention must be given to the design, methods, timelines, and personnel 
involved from the beginning of guidance program planning. The evaluation plan must 
clearly indicate what is to be evaluated and why, how, and when the evaluation will be 
carried out, and it must identify the responsibilities of all involved in the process. It 
must also provide formative data for ongoing corrections, summative data for future 
changes, and evidence to support continuing or expanded support for the guidance 
program. 
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Planning 

L'sing information from Chapters 15 and from references, the following is an example 
of a plan to evaluate a guidance program. 

Planning and Organizing 

1. Purpose of evaluation. Determine the purpose of the evaluation, because the 
purpose may dictate what is to be evaluated as well as how it is to be done. 
Example A. The purpose of this evaluation is to determine If students 
know how to use career information in selecting a career plan. 

This evaluation can be as simple as having students use information in determining 
steps to reach the career of their choice. Their use can be validated by teachers, 
counselors, or other adults. 

Example B. The purpose of this evaluation is to determine why qualified 
students do not enroll In advanced classes. 

This can be answered through structured interviews with parents and students 
separately or together coupled with a review of grades of students currently 
enrolled. 

Example C. The purpose of this evaluation is to determine the effect the 
guidance program has on students 1 classroom performance. 

Thi;; may require a review of student records to determine who had guidance 
instruction on learning, a review to see which teacher had consultation or inservice 
by counselors on classroom techniques, or possibly in-depth simetured interviews 
as well as a paper/pencil assessment. 

2. Audience. The audience for the report has a major impact on the questions, the 
data collected, and the way in which the report is prepared. 

Example D. In order to expand the elementary guidance program, impact 
data from the current program is needed for a report to the Board of 
Education. 

The data should include student results, staff and parent results, and any impact 
information such as higher test scores, fewer referrals, or less vandalism. 
Example E. The guidance staff needs to know If the parent effectiveness 
groups are having an effect on student attendance and behavior. 

Past and current attendance records, teacher observations, referral records, and 
discipline records of targeted students will provide the data needed by the staff in 
considering program changes. 

Example F. The superintendent has requested information on the 
effectiveness of group guidance in middle school. 

This audience will want information on what group guidance is going on and on 
what subject, and what impact this has on students' learning. Such a report might 
include recommendations centered on implementation of a teacher adviser 
program. 
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3. Questions. The questions must address the purpose of the evaluation as well as 
explicitly answer any questions fro;;, a specific audience. 

Example G. The principal wants to know, "How is the Career Center 
doing?" 

The questions "Do students know the location of the Career Center?" or "Do 
students know what is in the Career Center?" both seem inadequate to measure 
effectiveness. A set of questions such as those that follow provides more accurate 
formative data to answer the principal's question: "What is the GIS and how is it 
used?" "Name four resources i:i the Career Center and describe what information is 
in each." "Are the students' Educational and Career Planning Folders up-to-date?" 
"Does each eleventh grade student have an up-to-date resume?" 

4. Program audit. A program audit provides information as to whether the goals and 
objectives address the areas to be evaluated. The audit can be completed by 
comparing program elements against those in Figure 1 in Chapter 2 or Figure 2 in 
Chapter 3. The comparison will help to determine if your program is ready for 
evaluation and will help identify and correct missing or incomplete program 
elements. 

5. Needs assessment. It is important to review available needs data collected by 
decision makers. This may aid identification of areas of concern and/or aid data 
collection. 

6. Participants. Participants in r N evaluation process should be identified as early as 
possible and be involved in all planning. The relationships between project staff 
and evaluators (internal and external staff) and between guidance program 
evaluators and other district evaluation staff must be well defined. Equally 
important, the ongoing roles of parents, staff, students, and other community 
members must be clearly delineated. 

7. Evaluation team. Establish an evaluation team that has representative expertise in 
all areas pertinent to evaluating the entire project. 

8. Training. Identify training needs for the evaluation team and develop plans to 
complete training. 

9. Costs. Determine as closely as possible the cost of completing the evaluation and 
make a judgment as to appropriateness in the context of the total budget. 

10. Schedule. Develop a schedule of time lines for the tasks to complete the 
evaluation. For example: 

• Committee meets, reviews plans October 4 

• Committee develops evaluation design October 11 

• Committee completes detailed evaluation design October 18 

• Committee presents complete evaluation plan to 

superintendent's counsel for approval October 25 

11. Monitoring. Establish monitoring dates and the role of the monitoring team. This 
will include who will perform the function, frequency of reports, and to whom 
reported. 

12. Special issues. All special issues peuinent to the evaluation will be identified and, 
when possible, considered in the design and processes. 
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Evaluation Design 

1. Questions. Delineate questions to be answered. 

2. Data. Specify what information must be produced to satisfy what audience. 

3. Coordination. The questions and required information should be developed in 
conjunction with project staff, consultants, and other team members. 

4. Comparison. If possible, specify a comparison procedure to estimate what the 
participants' performances might have been without the guidance program. 

5. Sampling. Determine the sampling procedures. 

6. Data collection. Set procedures and time to collect data. 

7. Instrumentation. Determine the kinds of assessment instruments and strategies 
needed to collect the required information (structured interviews, observations, 
criterion-referenced or norm-referenced tests, self-reporting questionnaires/rating 
scales, attendance, GPA's, etc.). 

8. Data analysis. Determine 'inds of scores that will be most useful and most 
appropriate, and who will do the analysis. 

9. Written report. Decide on the form of the final report. 

10. Feedback. A means should be developed to allow persons affected by the 
evaluation results to review the evaluation findings, to provide feedback on 
identified problems, and to offer suggestions for modifications. These persons 
might be parents, advisory council, counselors, or administrators. 

Reporting Results 

1. Written report. The report of the evaluation includes a description of the project 
and objectives, how the evaluation was conducted, the findings, a discussion of the 
findings, a set of recommendations for program changer, or a number of options for 
adoption, and an Executive Sum mar/. 

2. Oral report. Materials must be developed for presentations unique to specific- 
audiences, i.e., for the Board of Education, administrators parents, community 
groups, and the guidance advisory councils. Who will present to what group must 
also be determined. 

Use of Findings 

1. Changes. Develop a system to collect input from various groups or suggest actions 
that might be taken to address specific recommendations. 

2. Follow-up. Develop a set of follow-up procedures with reporting dates to provide 
information on the progress of implementing the corrective actions. 
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Epilogue 



The Person-Oriented Approach in 
Evaluating Career Guidance 
Programs 

John O. Crites, University of Colorado 



The preceding content of this guide has centered on evaluating current 
guidance programs, utilizing the strategies, techniaues, and technology 
typically available within school districts. The following is a presentation of 
how guidance will be evaluated in the future— on a student-by-student 
basis. New technology portends easy, rapid, and accurate collection of 
data from each student. It will allow for automatic aggregation of data to 
provide information to ensure that all students acauire the necessary life 
skills to become contributing members of society. In some instances the 
following case study may already be a possibility, and where it is not 
currently feasible, the authors recommend that planners study individual 
evaluation as they chart new directions for guidance programs, 



The effectiveness of career guidance programs is a critical question on several levels- 
policy determination, program implementation, and personal satisfaction and success. 
This epilogue builds upon the collective conclusions of previous reviews (Crites, 1987) 
and extrapolates from them, as well as present theory and research and new models . 
and methods for evaluating career guidance programs, especially as they affect and 
influence the individual student. The focus of this approach is person-oriented, It 
recognizes and emphasizes the individual student, not only as a client, but also as the 
basic unit of analysis for any kind of program evaluation. The rationale for this 
viewpoint is that the effects of career guidance programs are not uniform and 
monolithic. Policy statements typically assume the uniformity myth, and programs often 
are designed accordingly, yet individual students may react quite differently to the same 
career guidance program. Whatever the reason for these differing reactions, the 
implication is that greater attention to individual differences among students must be 
given in both policy statements and program designs. In other words, choice of career 
guidance models and methods (Crites, 1981) becomes a consideration of what the 
most desired outcomes of the counseling are for this student and how these can best be 
achieved with him/her at this time. 



The Person-Oriented Approach to Career Guidance: Theory 

The central concept in this approach is what has been termed Career-By-Objectives 
(CBOs) (Crites, 1987). The analogue for CBOs in career counseling comes from the 
weil-establishcd practice of MBOs (Management By Objectives) in business and 
industry. Following reverse programming principles and procedures, the rationale for 
specifying MBOs is first to delineate what the desired outcomes of a program are and 
then to "work back" through each step that is necessary to achieve the objectives. Not 
only does this process establish an accountability criterion for the program, that is, the 
extent to which the objectives are attained, but it also eliminates through reverse 
programming unnecessary steps for goal attainment. Because goal- irrelevant steps are 
eliminated, the cost effectiveness of MBO programs is appreciably greater than more 
global approaches that are less specific and directed. Costs can be objectively defined 
by amount of time invested in the program as compared with others, the resources 
expended ("dollars and cents"), the number of personnel needed to achieve the 
objectives, and so on, all of which can be factored in as paiameters for a cost benefit 
analysis. Thus, inherent in MBO programs are the criteria for evaluating them, the 
parallel for assessing the effectiveness of career guidance programs being obvious. 

The concept of CBOs is new and needs to be articulated further to indicate how it is at 
the core of a person-oriented approach to career guidance. Following the rationale for 
MBO programs, the first step would be: What outcome(s) does this student want from 
his/her contacts with the career guidance counselor? Answers to this question may 
range from choice among career options to how a course of study relates to particular 
careers. Or, as is so often true of undecided students, the objective is to decide on an 
objective. The counselor and client work collaboratively on formulating CBOs that can 
be achieved within a given time frame, which is frequently a central consideration in 
career guidance because clients typically want to make a decision "right now." This 
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sense of urgency (or what Crites [1981] has termed the "immediacy imperative") must 
be built into the second step in CBO career counseling, which is to specify what has to 
be done to accomplish an objective, beginning with the next to last behavior or activity 
and reverse programming to the beginning of the career counseling prcxess. If 
contingencies arise in these behavioral acts in implementing the plan, then appropriate 
changes are made collaboratively by client and counselor, but they must be consistent 
with the CBOs. Also, if it becomes apparent during the guidance and counseling 
process that the objectives need rev ision, then new CBOs are delineated, along with 
steps necessary to attain them. Thus, this approach to person-oriented career guidance 
incorporates flexibility while it achieves specificity. 

To illustrate how reverse programming was used with an actual client, consider the case 
of Randy, a high school senior who came for career counseling because he was 
undecided. During the first interview, it became apparent that his primary CBO was to 
make a career choice, but that he had no idea of how to accomplish this objective, 
never having made a major life decision before. The counselor explained that before he 
could make a career choice he needed to relate his aptitudes and interests to those 
required by different occupations and identify those that were most realistic for him. To 
do this, Randy first needed to have information about himself and the world of work. 
The counselor pointed out, however, that to use this information in translating his self- 
concept into a compatible occupational role, he had to be career ready or mature 
enough to make a decision. Because he came for career counseling due to his 
indecision, it seemed important to assess his career readiness (career maturity). 
Therefore, Randy and the counselor agreed upon the following sequence of steps in the 
career counseling plan: 

• last step: declare career choices and chart career paths 

• fifth step: gather occupational information 

• fourth step: interpret test results 

• third step: take aptitude tests and interest inventory 

• second step: increase career readiness 

• first step: assess career readiness 

Reversing these steps, the counseling began with Randy taking the Career Readiness 
Inventory' (CRI) and progressing through the remaining steps to his CBO. 
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Figure E.1 
A systems approach 
to person-oriented career guidance 



A Systems Approach to Person-Oriented Career Counseling 

This highly abbreviated summary of the career counseling for Randy reflects a general 
schematic for career counseling based upon the systems approach shown in Figure E.l. 
There are three levels in the system, the topmost being Theory, which represents the 
distinction made previously between career choice process and career choice content 
(Crites, 1978). To the left are the prwess variables attittddes and competencies thai 
determine the readiness of an individual to make a career choice. Analogous to the 
concept of reading readiness, career readiness is a sine qua non for career decision 
making. If a student is sufficiently career ready (has mature attitudes and 
competencies), then he or she can make a realistic career choice, based on aptitudes, 
interests, and opportunities, as depicted on the right side of Figure E.l. This progressive 
"funneling" of career options, through what Tyler ( 1961) calls the "exclusion process," 
culminates in a career choice— and its implementation in job search and placement. 

On the Testing level of the system, the process and content components of career 
choice are given operational definition by inventories, tests, booklets, and forms that are 
used in the person-oriented (CBO) approach to career guidance. For example, The 
Career Readiness Inventory (CRI) (Crites, 1985b), a second generation revision of the 
Career Maturity Inventory (CMI) (Crites, 1978), measures the degree of career 
development of the student's choice attitudes and competencies. The CRI can be either 
hand or machine scored and the results interpreted with the Career Developer (Crites, 
1984), which also "teaches the test," so that by explaining the career-ready responses to 
the CRI, career maturity is significantly increased (Crites, 1985c). 

Given a sufficient threshold of career readiness, the student next takes aptitude and 
interest measures as a basis for career choice content. As shown in Figure F.l, there are 
different branches along which the testing can proceed. If assessment of previous 
learning in such areas as language, mathematics, and reading are necessary, then the 
client would take one of the many standardized achievement tests which are available, 
or the scores which may be in the student's cumulative record, e.g., from an 
achievement battery such as the Iowa Tests of Educational Development (ITFD), could 
be used. If there are below average areas on the achievement tests, remedial education 
may be necessary before administration of aptitude tests for career counseling. 
However, if achievement levels are adequate, then a good multi-factor aptitude battery 
can be administered (if not already available). 

In the system depicted in Figure F.l, the Career Decision Maker (CDM; see discussion 
below) reports and interprets the scores from the aptitude battery. Originally, the CDM 
was keyed to the General Aptitude Test Battery (GATB), on which there is the most 
extensive data available for defining and delineating Occupational Aptitude Patterns 
(OAP's). The GATB has been administered to senior high school students across the 
nation for many years. If GATB scores are recorded for a student, they can be used in 
the microcomputer system (see next section) to compare the student's aptitudes to 66 
OAP's which include all 12,000 occupations in the world of work and to identify those 
for which the student is qualified. 




Similarly, there is now available, and field tested, a form of the Career Decision Maker 
(CDM) that can be used with the Armed Services Vocational Aptitude Battery (ASVAB). 
Since the ASVAB is administered to over one million high school students each year, 
without charge, there is a great advantage economically to using it with the ASVAB/CDM 
form and the coordinated Career Path Finder (GPP) reference which can be used to 
chart both civilian and military career paths. The CDM not only prints out the ASVAB 
academic and occupational composites, but it also translates these into OAP's using the 
GATB database. The CPF can then be used with either the Guide for Occupational 
Exploration (GOE) reference or the computerized Career Information System (CIS) to 
gather occupational information on career options and paths. The Military Career Guide 
(MCG) reference is also keyed to the CPF to explore military careers, many of which 
may be cognate with the student's civilian career paths. 

Finally, if neither the GATB nor the ASVAB is available, the CDM can be used with 
selected tests of the Employee Aptitude Survey (FAS), which measures variables 
comparable to the GATB verbal, numerical, spatial, and clerical factors. Neither the FAS 
nor the ASVAB appraises the GATB manual dexterities, which require performance tests. 
The other two multi-aptitude batteries, the Differential Aptitude Tests (DAT) and the 
Knowledges-Skills Abilities (KSA) Survey, have not been linked to the CDM through the 
GATB database at the present time. The Career Decision Maker (CDM), on which the 
multi-factor aptitude scores and OAP's are printed from the microcomputer system, is 
designed along programmed instructional principles. It has four parts that systematically 
take the suident through the career choice process. 

The first enumerates the three steps in this prcx'ess, following the "Matching Model," 
beginning with "Step 1: Rank your INTERESTS," then proceeding to "Step 2: Know your 
APTITUDES," and culminating with "Step 3: Make your CHOICE." The second part 
defines the personal orientations and work environments in Holland's (1979) R1ASEC 
system and asks the student to rank them in order of preference from high ( 1) to low 
(6). The third part reports and interprets the scores from the multifactor aptitude 
battery, as well as relating them to predicted Levels of Educational Development (LED). 
Lastly, the fourth part collates the interest rankings and aptitude results with the OAP's 
printed in the squares of the "Career Choice Chart." On this chart, the student outlines 
in pen or pencil the sequence which corresponds to his/her highest three ranked 
interests (or Career Fields) at the attained Career Level defined by the general aptitude 
score. Within the squares are the OAP's for which the student qualifies. Here, then, 
brought together in one place is the student's vocational self- concept, based upon the 
best and most comprehensive objective test information available. 

At this point in the overall prcxess, additional tests (Figure E.l) may be administered to 
provide supplementary information on a student's capabilities. Although ranked or self- 
expressed interests are as good or better predictors of eventual occupation-engaged in 
(Holland and Lutz, 1968: Borgen and Silling, 1978; Dolliver, 1969) than standardized 
interest inventories, the latter may be useful with clients like Ranch', who are undecided 
or indecisive (Crites, 1981). Furthermore, additional assessments maybe needed to 
appraise special talents, such as artistic and musical aptitudes, as well as gross and fine 
manual dexterities. Scores from such measures can be integrated with OAP's printed in 
the squares of the CDM. 
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The next (and last) stage in the career choice process leading to career choice content 
is to translate the student's vocational self-concept (aptitudes, interests, etc.) into 
compatible occupational options This objective is accomplished by means of the 
Career Path Finder (CPF), also constructed along programmed instructional principles. 
In a stepwise fashion, it takes the student from a SURVEY (Step 1) of the world of work, 
to a narrowing of CHOICE(S) (Step 2\ finally to a charting of a primary 7 and secondary 
career PATH (Step 3) over the next five years. In the SURVEY part of the CPF, the 
student records the information from the "Career Choice Chart" of the CDM on what 
amounts to the world of work— a listing of all the GATB 66 work groups cross-classified 
by career fields and levels. Indexed to each of these work groups are the corresponding 
sections of the Guide for Occupational Exploration (GOE), which provides current 
descriptions of the work performed, capabilities needed, and opportunities in the field. 

From this survey, the student then selects six occupations, using the Career Choice 
Checklist in the CHOICE part of the CPF. Following an elimination process, the student 
progressively narrows these choices to two, one designated primary and the other 
secondary. These are projected in the last part of the CPF (PIAN) over the next five 
years as career paths, using the completed example for a career in computer 
programming as a model. Thus, the student systematically arrives at two concrete, "in 
hand" career paths from an objective assessment of personal characteristics and a 
matching of these with those required by occupations in the world of work, using 
dex'umented information. But probably even more important, the student learns general 
decisional skills by following the stepwise format of the CDM and CPF, which research 
evidence indicates generalizes to other areas of life decision- making. 

This systems approach to career guidance has been made operational by the increasing 
availability and use of microcomputers. If it were not for the recent developments in 
synchronized computer and peripheral technology, the systems approach to career 
guidance would not be possible. However, there are now three components available 
that make the system a reality and have made possible its field testing during the past 
five years. The first component is a National Computer System ( NCS) Sentry 3000 
Scanner, through which aptitude and interest answer sheets can be prcxessed and 
scored by the computer in a matter of seconds. The second component, a personal 
computer (e.g., IBM PC XT), not only scores the aptitude tests and interest inventories, 
but also through a specifically developed software program (Crites, 198*>a) matches the 
student against all 12,000 occupations in the economy and prints out the realistic career 
options on the CDM in less than a minute. Thus, the technology accomplishes in a 
fraction of the time the profile or psychograph (Crites, 1969) comparisons that took 
counselors, using inspect ional subjective techniques, many hours to complete. The 
software can also be used on the mainframe when assessment scores are uploaded to 
gain greater macro capability. If a scanner is not available, scores from the aptitude tests 
can be key entered in the microcomputer, although this is a sizable task when there are 
large numbers of students— and the error rate is greater than with the scanner. 
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Evaluation Models and Methods 



Using the storage and retrieval capabilities of the microcomputer, a variety of data on 
each student can be processed to assess goal attainment (career counseling success) 
and to aggregate outcomes across students for the evaluation of career guidance 
programs in schools and districts. Primary data for such analyses are: (a) the goal 
attainment scale score, (b) client attributes (demographic and/or psychometric), and 
(c) type of "treatment" or approach used in the career counseling. Other significant 
parameters that can be factored in are (a) number (if client contacts, (b) length of each 
contact, (c) counselor/client gender, (d) individual and/or group guidance, and so 
forth. All data can be entered into the microcomputer either on scannable forms or by 
manual key entry by the counselor or a clerk. A file is created in the computer for each 
student that can be manipulated for any number of relevant statistical analyses. These 
will vary, of course, from one program to another, but the basic evaluation software can 
be uniform for any situation in which there is an interest in accountability and cost- 
benefit analysis. Descriptive statistics on program activities for administrative purposes 
can also be generated as can cumulative career records. Appropriate software is 
currently being developed for these analyses, which require no statistical knowledge or 
training by the counselor. 

To illustrate the application of this process, further description of the goal attainment 
scale is necessary. Approximately 20 years ago, goal attainment scales (GAS) were 
proposed by Kiresuk and Sherman ( 1968) to provide a criterion for evaluating mental 
health programs. The rationale for GAS is that this prcxedure can be used whatever the 
goals or objectives of counseling are, as has been proposed for the definition and 
measurement of vocational success (Crites, 1969). Much as their name connotes, goal 
attainment scales ask the client and/or counselor how much the counseling has 
achieved the goals initially set at the beginning of the counseling. In other words, to 
what extent has the goal(s) been attained— none at all, somewhat, closely, or 
completely. For career counseling/guidance, a Likert-type unidirectional scale, like the 
one shown in Figure F.2, can be used. Note that the Career By Objective (CBO) which 
us agreed upon by client and counselor at the outset of the career counseling/guidance 
is entered on the form where it says "CHO." Then, when the career counseling/ 
guidance has been completed, using separate forms, the client and counselor rate how 
closely the CBO has been attained. 

CBO: 



To what extent has this objective been achieved through career guidance? Place a check (\/) on the 
line to indicate degree of goal attainment. 

little mostly or 

or none completely 
(1) (2) (3) (4) (5) 



Figure E.2 

Career-By-Objective (CBO) Goal Attainment Scale 




The great advantage of this procedure is that different goals or objectives can be 
measured on the same scale, which means that the goal attainment by one client can be 
directly compared with that of another client even though their goals or objectives were 
different. In this way, GAS scores can be aggregated across clients and analyzed 
statistically, using the same metric, for any number of "break outs/ 1 such as counselor 
differences, individual vs. group career guidance differences, grade/school differences, 
etc. Analysis can also be made of the relationships (correlations) of GAS with individual 
difference variables, such as achievement and gender. Thus, the systems approach to 
person oriented career guidance can evaluate how effectively programs assist 
individuals in the attainment of their career goals and how the results impact and 
suggest revisions in policy statements (Crites, 1987). 
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Needs assessment in a results-based guidance program is analogous to needs 
assessment in any other competency-based curriculum program. The mathematics 
program, for example, does not base its curriculum or course of study on students' 
opinion of their need for math skills. Nor does it first test students, identify what they 
do not know, and address the curriculum only to those deficiencies. In order to 
ascertain what students do not know it is first necessary' to establish what they should 
know. Learning expectations or standards must be in place prior to identifying such 
deficiencies or gaps. 

In the mathematics program as in all school curriculum areas (including guidance) 
professionals identify desired results and develop and implement a program 
(curriculum). Needs data and subsequent program change and improvement come 
from measuring students' knowledge, attitudes, and skills in relation to what is 
expected of them at a specified time or level. 

In Chapter 2, the collection and use of needs data constitute the eighth element in the 
results based guidance program. In that program, needs are viewed not as perceived but 
rather as actual performance gaps. The collection and use of such needs data are a part 
of the continual program evaluation and improvement and do not precede program 
design and implementation. 

Typically and historically counselors have used needs assessment as the first step in 
program planning. Most process based programs continue to use needs data in this 
manner. That is, perceived needs of various groups to be saved by the program are 
surveyed and the program is designed to react to the desires of the various groups. The 
different perceptions among the groups (students, teachers, parents, administrators) are 
noted and decisions are made to accommodate these differences in planning and 
scheduling various resources and activities. 

Needs data and their use will van- with the reasons for the needs assessment and the 
definition of need used within the guidance program. Several types of needs assessment 
are currently used in guidance programs. 

In a results-based or competency based program "needs" are defined as the gap or the 
difference between actual and desired results or competencies. Needs assessment is the 
measurement and/or verification of these differences. If other types of needs data are 
collected and used, they must serve as a supplement to these results-based needs. 

In order to provide additional data to, or involve additional audiences in the guidance 
program, some programs, in moving from a serv ice- or prcxess-based model to a 
results-based model may have a desire (and ample time and resource? ) to include, in 
addition to an "actual" needs assessment, a "perceived" needs assessment. A brief 
description of various types of needs assessments are provided to assist programs in 
collecting data based on both actual competency needs and perceived competency or 
process needs. 
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Type I 

Needs Based on Competency Demonstration 

A results-based program is inductively developed first, prior to the assessment and use 
of student needs data. After the program has been designed and all elements 
implemented, needs assessment seeks to determine the gaps or the dfference between 
predetermined, expected, or desired results and the actual immediate, intermediate, 
and/or long range results (knowledge, attitudes, and skills) that students, teachers, 
and/or parents have acquired. Only after the expected or desired results have been 
identified by the professional staff and a program is in place, is it necessary' to assess the 
present level of competencies in the determined areas. If, for example, it is found after 
testing, observation, etc., that all students are currently meeting expectations for a 
particular result, then, at this time, for this result, and for this particular group there is 
no need for program change. In all probability resources and program activities 
addressing this expected result would be continued at their present level. 

If students are not performing as expected or cannot demonstrate competencies in 
specified areas, however, different strategies or resources may have to be focused on 
this need. This form of needs assessment clearly aims at identifying the level of 
competence in relation to predetermined criteria or standards. Without directly 
addressing current strategies, resources, and program components, or identifying new 
ones, this type of needs assessment can clearly provide evidence that change or 
improvement is necessary to eliminate the gaps between desired and actual results. The 
severity of the deficiency or gap along with other considerations can also provide 
direction to staff in determining change and program priorities. When combined with 
program audit results, prcxess and resource evaluation, and other product evaluation 
data, needs identified in this manner provide direction for program improvement. !n 
essence this type of needs assessment is a part of the formative evaluation procedures. 

rnlike the preceding program, The Comprehensive Guidance Program for Texas Public 
Schools ( 1991), while using a competency-based approach to guidance programs, 
establishes the assessment of student and community needs as the center of the local 
program development process. It emphasizes that the design of the guidance program 
should be based on a knowledge of the students' and community's needs and a 
consideration of current and potential resources available. Needs data are grouped into 
three categories: (1) demographic data (mobility, rates, language, socioeconomic scale, 
immigration patterns, etc.); (2) student achievement data (test scores, masters' of 
essential elements, dropout rate, promotions/retentions, student grades); and (3) 
student conduct data (discipline, attendance, school participation). 

These data are then used to determine the quantitative (quantity of resources to devote 
to each program component) design of the program and the qualitative design which 
includes a description of each component and the priorities in each component; the 
content of the program including competences and priorities for each grade level; and 
the populations to be served such as students, teachers, administrators, and parents, and 
the priorities for each. 
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Type II 

Needs Based on Student, Teacher, and Parent Opinions of 
Competency Achievement or Deficiency 

Gysbers and Henderson ( 1988) see this type of needs assessment as a useful tool in the 
program improvement process but do not recommend it as an initial task. 
Although this assessment can be done first, we do not believe the results will be as 
useful as those of one done during this phase of the program improvement process. 
A major reason we recommend waiting until this phase is that it is not until now that 
you have identified the student competencies for your program. And, in the type of 
assessment we are proposing, student competencies statements become, in effect, 
the needs assessment items. In fact, this part of the designing process could just as 
well be called our inventory of student competencies— an inventory of where 
students are in competency development and where they would like to be in their 
competency acquisition. ( 166) 
The difference between the first type of needs assessment (competency demonstration) 
and this type (student opinion) can be noted in the fact that in this second type of 
assessment students arc asked to give their opinion of their current status (not 
demonstrate it), and their opinion of their expected level of competency (not 
predetermined standards by staff). These needs data are based on what students think 
they can do compared with what they wish they could do. Without further efforts to 
substantiate them, they are usefiil but unverified. An example of this type of needs 
assessment is: 



To fill out this questionnaire, students were asked to complete the following steps. 

Step 1: Read each sentence carefully. Each one describes things students are able to do in order to 
demonstrate learning in that particular area. 

Now make a decision. Are you able to do what the sentence describes? Are you unable to do what the 
sentence describes? 

Fill in the circle that shows what you think. 

If you think you are able to do what the sentence describes. . . 

fill in circle a ® ® © 

If you think you are not able to do what the sentence decribes. . 

fill in circle b ® ® © 

Step 2: Choose the five sentences on each page that describe what you would really be interested in learning to 
do. Some of the statements will really interest you and some will not In the second column (circles are 
lettered c), fill in the circle for each of the five statements that you feel you need to learn how to 
accomplish. 

For example: 

I CAN _ . 

1 . evaluate the importance of having laws and contracts to protect producers. ® 4P © 

2. provide examples of decisions I have made based on my attitudes and values. ® (B> © 
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Type III 

Needs Based on Opinions Concerning Importance of 
Competencies 

Another form of needs data collected by competency-based guidance programs 
addresses the importance attached to the competencies or results expected from the 
program. This type of data is usually collected in combination with other types of needs 
data. The following example from the Florida Department of Education is based on a 
similar instrument found in the National Occupational Information Coordinating 
Committee's Implementation Guide and is an example of such a combination in that it 
collects opinions on both importance and achievement of competencies. 



Career Development Needs Assessment 
Sample Instrument 

POSITION: Administrator Teacher Counselor 

Student Parent Other 

GRADE LEVEL: 
INSTRUCTIONS: 

We are conducting a survey of the career development needs of our students. This survey lists attitudes, skills, 
and knowledge that are related to effective career awareness and exploration. Read each item and give it two 
different ratings: (1) Importance— for students in our school, and (2) student achievement— your perceptions of 
how many students in our school already have the attitude, skills, or knowledge. 

Circle your choice using the following Importance scale: 

Of Great Of Little 

Importance Importance 



1 



Circle your choice using the following student achievement scale: 



No Students Few Some Most All Students 

0-10% 11-30% 31-70% 71-90% 91-100% 



Competency 1: Knowledge of the influence of a positive self-concept on career development 

1. Students can access personal likes and dislikes. 

Importance Perceived Achievement Total 
54321 54321 

2. Students can assess individual attributes required for successfully fulfilling different roles. 

Importance Perceived Achievement Total 
54321 54321 



The above example asks respondents to assess needs ( importance and achievement) in 
the third person. That is, students, administrators, teachers, etc. are asked to respond for 
students in general, not for their own personal needs. This type of needs survey also 
could be changed to address a student's opinion of his/her own achievement and the 
importance he she attaches to a competency. 
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Type IV 

Needs Based on Opinion of Resources, Strategies, and Services 
Provided In the Guidance Program 



This fourth type of needs data can b e collected as a part of a follow-up study as well as 
ongoing needs assessment surveys. This type of needs assessment solicits student, 
teacher, or parent opinions concerning school programs and resources. It indirectly 
identifies perceived needs by asking respondents to rate their degree of satisfaction 
with various features and services of the guidance or total school program. The 
following samples provide examples of this indirect approach. 



> 



> 




1. School district transportation system 
(buses, etc.) 


0 


0 0 0 0 0 


2. Guidance services (counseling, 
college planning, etc.) 


0 


0 0 0 0 0 


o, opecidi tscauBiTifc programs 

(accelerated/gifted ed., special ed.. etc.) 


0 


0 0 0 0 0 


4. Special tutorial assistance (remedial 
reading, tutors, etc.) 


0 


0 0 0 0 0 


5. Sports-related activities 


0 


0 0 0 0 0 


6. Fine arts activities (band, theater, art 
exhibits, etc.) 


0 


0 0 0 0 0 


7. School-sponsored social activities 
(dances, etc.) 


0 


0 0 0 0 0 


8. Student organization? (science club, 
FFA, pep club, etc.) 


0 


0 0 0 0 0 


9. Textbooks and instructional materials 
used in your classes 


0 


0 0 0 0 0 


10. Variety of courses offered by this high 
school 


0 


0 0 0 0 0 


11. Practical work experience offered by 
this high school 


0 


0 0 0 0 0 


12. Flexibility to select your own classes 


0 


0 0 0 0 0 


13. Quality of classroom instruction 


0 


0 0 0 0 0 


14. Testing /grading procedures 


0 


0 0 0 0 0 


15. Attitude of the teachers/administrators 
toward students 


0 


0 0 0 0 0 


16. Availability of teachers outside of 
class hours 


0 


0 0 0 0 0 


1 7. Availability of administrators 


0 


0 0 0 0 0 


18. Class size 


0 


0 0 0 0 0 



Excerpted from High School Follow-up Survey, ©The American 
College Testing Program (ACT). 
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1. Quality of classroom instruction 


0 


0 0 0 0 0 


2. Grading policies and procedures 


0 


0 0 0 0 0 


3. Variety of courses offered 


0 


0 0 0 0 0 


4. Amount of homework assigned 


0 


0 0 0 0 0 


PloviHilitv tn colopt unur nuu n crhaHi il a 
riOAiuiiiiy iu ociui/i yuui uwm ounuuuit? 

of courses 


0 


0 0 0 0 0 


fi Tovthnrtkc anH inctriir*tinnal ma tori ale 

used in your classes 


0 


0 0 0 0 0 


7 MiimHor anH ItinHc nf to etc nit/on 


u 


n c\ n n n 
u u u u u 


8. Requirements for graduation 


0 


0 0 0 0 0 


9. Opportunities to take part in 
extracurricular activities 


0 


0 0 0 0 0 


1H MiimHor anH ItinHc nf ctnHont 

1 v. WUIIIUd allU nlMUD UI OlUUOMl 

organizations 


0 


0 0 0 0 0 


1 1 Amount nf Amnhacic nlapoH nn 
i i • miiuuiu ui crniuiiaoio uiauuu um 

extracurricular activities 


0 


f\ f\, f\ f\ f\ 

0 0 0 0 0 


19 Qphn/^l^cnnncrtrAH cn^ial ouonlc 
l£i ul/l IUUI OUUI I ouicu ouuiai OVOlllO 

(dances, etc.) 


0 


0 0 0 0 0 


1ft Rolinimic APtivittAC anH nrnnnmc 
iw. ncnyiuuo auiivuivo anu uiuyicsno 


u 


n n n n n 
u u u u u 
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14. Availability of counselors to students 


0 


0 0 0 0 0 


1 5 Value of information provided by 
counselors 


0 


0 0 0 0 0 


16. Assistance provided in course 
selection and class scheduling 


0 


0 0 0 0 0 


17. Counseling provided for personal 
problems 


0 


0 0 0 0 0 


18. Assistance provided in selecting and 
planning a career 


0 


0 0 0 0 0 


19. Assistance in finding a full-time job 
after graduation 


0 


0 0 0 0 0 


20. Assistance with college planning and 
selection 


0 


0 0 0 0 0 



Excerpted from High School Student Opinion Survey, ©The American 
College Testing Program (ACT). 
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Type V 

Needs Based on Requests for Assistance 



This fifth type of needs assessment is a more direct method in that it usually allows 
students to specifically identify the program activity or component in which they would 
like additional staff effort. This type of needs survey differs from previous types in that it 
implies a personal deficiency or dissatisfaction by directly translating their need into 
suggested program change or improvement. Two examples of survey items for this type 
of needs assessment follow. 

SECTION III (Continued) 



> 



This item is NOT IMPORTANT or DOES NOT APPLY to me. 
want NO FURTHER help with this item. 

I want a LITTLE more help with this item 

want a MEDIUM AMOUNT of added help with this item. 
I want MUCH MORE help with this item, 
t 



0 0 


0 0 0 


/Zs dealing with the problems of divorce and/or 
^->^ parent separation in my family 


0 0 


0 0 0 


/g> dealing with pregnancy and/or being a 
v-*' parent while still in high school 


0 0 


0 0 0 


(62) learning how to stay mentally healthy 


0 0 


0 0 0 


(63) handling things that worry me 


0 0 


0 0 0 


/Ov handling pressure from my parents, teachers, 
\~s and other adults 


0 0 


0 0 0 


(66) dealing with feeling lonely and/or depressed 


0 0 


0 0 0 


controlling the stress or fear 1 feel before or 
kBs when I'm taking a test 


0 0 


0 0 0 


o learning more about suicide— the causes. 
v?J/ warning signs, and intervention techniques 


0 0 


0 0 0 


zO| knowing where 1 can go to get help with my 
viS/ personal problems 


0 0 


0 0 0 


maintaining and/or improving my physical 
health (exercise, nutrition, etc.) 


0 0 


0 0 0 


(70) understanding my own sexual development 


0 0 


0 0 0 


learning more about AIDS and other sexually 
transmitted diseases 


0 0 


0 0 0 


dealing with the effects of alcohol, nicotine, 
\ls and other addictive drugs 



From items 37-72 (on pages 3 
and 4\ select the 3 personal 
area? with which you MOST 
WANT ADDITIONAL HELP. 
Write the item numbers of those 
three in the circles to the right 



ooo 



Excerpted from Student Needs Assessment Questionnaire, ®The American 
College Testing Program (ACT). 
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11.1 would like to know more about 

A. What can help me in school to get better grades 

B. What can help me enjoy my free time 

C. What can help me earn more money now 

D. What can help me choose a career 

E. What can help me have more friends 

F. I do not need help at this time to know more about myself 

13. In planning my career (life's work) for the future, at this time, I need the most help f n finding out: 

A. What career fields there are (kinds of jobs available) 

B. What education is required for certain jobs 

C. What people do in different jobs (e.g., doctor, store manager, truck driver, etc.) 

D. How much a job pays 

E. Where jobs are found 

F. What are different ways of getting certain jobs 

G. Which jobs may or may not be needed in the future 

H. I do not need help at this time in planning my career 

19. I would like additional help in learning: (Select ONE or TWO of the following.) 

A. What kinds of jobs are available to students my age 

B. How much part-time jobs pay 

C. Where I can look for part-time and summer jobs 

D. How I make out an application form 

E. What the proper way is to interview for a job 

F. How I get a social security card 

G. Where I get a work permit 

H. I need no help at this time 



Excerpted from Educational and Career Planning: Individual Guidance Assessment, D. Johnson and 
S. K. Johnson. 



Summary 

The results-based program suggested in this guide is inductively developed prior ;o the 
assessment and use of student needs. After the program has been designed by 
professionals and/or recognized specialists and based on research and professional 
knowledge, needs data are collected from students to identify the gaps between desired 
and actual student attainment of competencies. The demonstration of competencies is 
the basis to determine if the program is delivering what it was planned to deliver. 

The essential type of needs data collected under this approach is used primarily to 
monitor and adjust the prexesses to ensure that off students acquire the desired 
competencies. Needs data arc, in essence, evaluation data. 

Other types of needs assessment can be used in addition to the results based type if 
uses of such data can be justified to supplement and complement the uses of the 
results-based needs. Several examples of instruments used to collect these 
supplementary needs data were presented and described. 



9 

ERLC 



106 

104 



References 



American College Testing (undated). High school follow-up survey. Iowa City: author. 
American College Testing (undated). High school student opinion survey. Iowa City: 
author. 

American College Testing (undated). Student needs assessment questionnaire. Iowa 
City: author. 

Florida State Department of Education (1989). Florida's guide to career development: 
career development program guide. Tallahassee: author. 

Gysbers, N. C, and Henderson, P. ( 1988). Developing and managing your school 
guidance program. Alexandria, VA: American Association for Counseling and 
Development. 

Johnson and Associates (1985). Individual guidance assessment. Irvine, CA. 

Texas Education Agency (1991). The comprehensive guidance program for Texas public 

schools: A guide for program development pre-K- 12th grade. Austin, TX: author. 



9 

ERIC 



107 



105 



Appendix 
B 



References for Evaluation Resources 



If 6 

ERIC 



How To's 



Adams, Kay Angona and Walker, Jerry P. ( 1979). Improving the accountability of career 
education programs: Evaluation guidelines and checklists. Columbus, OH: National 
Center for Research in Vocational Education, R&D Series 168. 
Alkin, Marvin C; Kosecoff, Jacqueline; Fitz-Gibbon, Carol; and Seligman, Richard 
(1974). Evaluation and decision making: The Title VII experience. Los Angeles: 
University of California. 

Anderson, Scarvia B.; Ball, Samuel; Murphy, Richard T.; and Rosenthal, ElsaJ. (1973). 
Anatomy of evaluation: Important concepts and techniques in evaluating education 
framing programs. Princeton, NJ: Educational Testing Service. 
Aurras, Philip A. and Dowd, Richard A ( 1966). A guide to assessment and evaluation 
procedures. Providence, Rl: The New England Educational Assessment Project. 
Bansherg, William H. and Warren, Toni McCarty ( 1986). Illinois counseling and 
guidance by objectives handbook. Normal, IL: Guidance by Objectives Vocational 
Research Projects, Illinois State University. 

Bard, Harold R. and Cody, John J. (1975). Minimizing measurement concerns in 
guidance evaluation. Measurement and Evaluation in Guidance, #(3), 175 179. 
Bissell, Joan S. (undated). Program evaluation as a Title VII management tool. Los 
Alamitos, CA: Southwest Regional Laboratory. 

Boruch, Robert F. and Cordray, David S. (1980). An appraisal oj educational program 
evaluations: Eederal, state and local agencies. Evanston, IL: Northwestern University. 
Bubenzer, Donald and Strother, Jo Anna (1985). Programmatic definition for guidance 
and counseling: 4.0 staff ing for programs. Charleston, WV: West Virginia Department of 
Education. 

California State Department of Education (1989). A guidance and counseling planning 
guide for developing quality student support services grades 9-12. Sacramento, CA: 
author. 

Campbell, Patricia B. (1987, Fall). Using students to perform sch(X)l program 
evaluations. Equity and Choice. Boston, MA: Institute for Responsive Education, 43 45. 
Campbell, Patricia B. ( 1988). Empowering students to perform sch(X)l program 
evaluations. Education Digest, April, 20-22. 

Carroll, Marguerite R, Ed. ( 1980). Standards for guidance and counseling programs. The 
School Counselor, 28(2), 81-86. 

Center for Vocational and Technical Education ( 1972). Developmental program goals 
for the comprehensive career education model. Columbus, OH: author. 
Cincinnati Public Schools ( 1981 ). Community survey model for school districts: 
Procedural guide. Charleston, WV: Appalachia Educational Laboratory, Occasional Paper 
008. 

Commonwealth of Virginia Department of Education (1984). Career guidance and 
counseling. Richmond, VA: author. 

Dayton, Charles W.; Helliwell, Carolyn B.; Richins, Duane: Diaz. Joe; and Trapnell, Fred 
(undated). Determining program goals and objectives: A systematic approach to 
guidance. Mesa, AZ: Mesa Public Schools. 



9 

ERIC 



in 

107 



9 

ERLC 



Education Daily. (November 15, 1984). 17, (222). 

Erickson, Richard G. (1985). Evaluate guidance activities. Columbus, OH: The National 
Center for Research in Vocational Education. 

Fink, Arlene and Kosecott, Jacqueline ( 1980). How to evaluate education programs: A 
compilation of ideas and methods that work Washington, DC: Capitol Publications, Inc. 
Franchak, Stephen J. ( 1981 ). Evaluating employer satisfaction. Columbus, OH: National 
Center for Research in Vocational Education, Research and Development series No. 210. 
Gill, Stephen Joel and Schumaker, Julie A. (1987). The role of the group worker in 
program evaluation, Journal for Specialists in Group Work, March, 43-48. 

Harrison, Laurie (undated). Specifying student performance objectives. Palo Alto, CA: 
American Institutes for Research. 

Helvorsen, James and Paden, Jon S. ( 1976). Swvey of effective school practices: Scoring 
guides 1 and 2. Dayton, OH: Institute for Development of Educational Activities. 
Imboden, N. ( 1978). A management approach to project appraisal and evaluation. 
Paris: Development Centre of the Organization for Economic Co-Operation and 
Development. 

Kaufman, Roger and Stone, Bruce ( 1983). Planning for organizational success: A 
practical guide. New York: John Wiley and Sons. 

Kaufman, Roger ( 1988). Planning educational systems: A results-based approach. 
I-ancaster, PA: Technomic Publishing Co., Inc. 

Kester, Ralph J. ( 1979 ). Using systematic observation techniques in evaluating career 
education. Columbus, OH: National Center for Research in Vocational Education, No. 
169. 

Law, Alexander I. and Bronson. William H. (1977). Program evaluatofs guide. 
Princeton, NJ: Educational Testing Service. 

Malak, Sharon; Spirer, Janet E.; and Land, Brenda Pellegrini (1979). Assessing 
experiential learning in career education Columbus, OH: National Center for Research 
in Vocational Education, Research and Development Series No. 165. 
McCaslin, N. L and Walker, Jerry P. ( 1979). A guide for improving locally developed 
career education measures. Columbus, OH: National Center for Research in Vwational 
Education, Research and Development Series No. 167. 

Mesa Public Schools (undated). Evaluation: Concepts and procedures. Mesa, AZ: author. 

Mesa Public Schools ( undated). Evaluation: Concept* tnd procedures VIII. Mesa, AZ: 
author. 

National Center for Research in Vocational Education (1972). Developmental program 
goals for the comprehetisive career educational model. Columbus, OH: author. 
National Center for Research in Vocational Education ( 1978). Evaluate your vocational 
program— module A- 11. Columbus, OH: author. 

National Study of School Evaluation ( 1979). Arlington, VA: Nation .1 Study of School 
Evaluators. 

National Study of School Evaluation (1979). Middle school/junior high school evaluative 
criteria, revised edition. Falls Church, VA: author. 

Phiilips, Gary and McColly, Bill ( 1 981 ). Challenge education: Background information 
for secondary school improvement project. Indianapolis, IN: Lawrence North High 
School. 

us 



Plakos, Marie; Plakos, John; and Bobcock, Robert (1977). Workbook on program 
evaluation Sacramento, CA: California State Department of Education. 
Ratzlaf, Leslie A., Editor (1987). The education valuator's workbook: How to assess 
education programs—volume I. Alexandria, VA: Education Research Group. 
Sortland, Robert A.: Snyder, Sally J.; McFarland, Joseph; and Webster, Barbara (1986). 
Professional development guidelines for secondary school counselors: A self audit. 
Alexandria, VA: American School Counselor Association. 

Wilson, Pamela J. (1986). School counseling programs: a resource and planning guide. 
Madison, WI: Wisconsin Department of Public Instruction. 

Young, Malcolm B. and Schuk, Russell G. (1975). Evaluation and educational decision 
making: A functional guide to evaluating career education. Washington, D.C.: 
Development Associates, Inc. 



Uses of Data 

Bissell.Joan S. (1980). Program evaluation as a Title VII management tool. Los 
Alamitos, CA: Southwest Regional Laboratory. 

Rudowski, Catherine W. (1988). 1987-88 annual report on the guidance and testing 
development. Upper Arlington, OH: Upper Arlington City School District. 



Evaluation Reports 

Crites, John O. (1987). Evaluation of career guidance programs.- Models, methods, and 
microcomputers. Columbus, OH: National Center for Research in Vocational Education. 
Ebert, Mary K. ; Richardson, William M. ; and Stevenson, Jose W. (1985). A study of the 
guidance program and its management in the Montgomery County public schools. 
Rockville, MD: Montgomery County Public Schools. 

Evaluation of elementary counsc'ing and guidance programs in Kentucky (1989). 
Conducted by the Kentucky Association of School Administrators and the Staff at the 
Appalachia Educational Laboratory. 

Flanagan, John C; Tiedeman, David V; Willis, Mary B. ; and McLaughlin, Donald H. 
(1973). The career data book: Results from Project Talents 5 year follow-up study Palo 
Alto, CA: American Institutes for Research. 

Jacobson, Thomas J. (1975). A study of career centers in the state of California. La Mesa, 
CA: Grossmont Union High School District. 

Jacobson, Thomas J.; Johnson, C. D; and Johnson, Sharon K. (1979). Youth Employment 
Program evaluation. La Mesa, CA: Grossmont Union High Schcx)l District. 
Johnson, C. D. and Mitchell, Anita M. (1974). Career guidance evaluation. Columbia, 
MD: Howard County Public Schools. 

Johnson, Sharon K. (1986) Evaluation of guidance: End of year report. Columbia, MD: 
Howard County Public Schools. 



113 

in,9 



Johnson, Sharon K. (1986). Evaluation of guidance: End of year report. Columbia, MD: 
Howard County Public Schools. 

Johnson, Sharon K. (1988). Design for guidance evaluation. Ellicott City, MD: Howard 
County Public Schools. 

Johnson, Sharon K. (1988). Results based guidance evaluation. Columbia, MD: Howard 
County Public Schools. 

Jones, G. Brian; Nelson, Dennis E.; Ganschow, Laurie H.; and Hamilton, Jack A. (1971). 
Development and evaluation of a comprehensive career guidance system: Final report. 
Palo Alto, CA: American Institutes for Research. 

Mitchell, Anita M. (1979). Assessment of state plans for career education: Final report. 
Los Alamitos, CA: Southwest Regional Laboratory. 

O'Neal, Sandra (1982). K- 12 guidance/ counseling program district evaluation report. 

Albuquerque, NM: Albuquerque Public Schools (ED 231 860). 

Peterson. Sarah and DeGracie, James S. (1983). Secondary counseling services as 

perceived by selected publics. Mesa, AZ: Mesa Public Schools. 

Vetter, Louise and Sechler, Judith, Ed. (1987). High school career guidance programs 

that work: Seven case studies. Columbus, OH: National Center for Research in 

Vocational Education. 

White, Robert M. (1986). Program evaluation: Counseling and guidance guaranteed 
services. Portland, OR: Jefferson High School. 



Evaluation Instruments 



Altschild, James W. and Ross, Melinda (1973). Career preparation. Columbus, OH: 
National Center for Research in Vocational Education. 

American Institutes for Research (1973). Intervieu* guide. Palo Alto, CA: author. 
Burry, James; Herman, Joan; and Baker, Eva L. (1984). A practical approach to local test 
development. Resource Paper No. 6. Los Angeles, CA: Center for the Study of Evaluation, 
University of California, Los Angeles. 

deVaurs, Nancy ( 1978). Implementation of student competencies in guidance/ 
counseling. San Mateo, CA: San Mateo Union High School District. 
Hamilton, Richard L. ( 1983). 1982 survey of drug abuse among Maryland adolescents: 
General report. Baltimore, MD: Maryland Department of Health and Hygiene Drug 
Abuse Administration. 

Hunt, Larson; Schrankel, Elizabeth J.; and Phillip, Stephen (1986). A proposal for 
community involvement in public school planning. Mesa, AZ: Mesa Public Schools. 
IOX Assessment Association (1982). A program evaluation handbook for health 
education program in alcohol and substance abuse. Culver City, CA: author. 

Johnson, C. D. and Johnson, Sharon K. (1979). Youth Employment Program evaluation. 
Grossmont, CA: Grossmont Union High School District. 

Johnson, C. D and Mitchell, Anita M. (1974). Career guidance evaluation ( for 13 and 
17 year old students). Hartford, CT: Connecticut State Department of Education. 



9 

ERLC 



114 

no 



Jones, Brian G. (1973). Interview guide: Counseling guidance and placejnent 
assessment project. Palo Alto, CA: American Institutes for Research. 
MaceMatluck, Betty J. ( 1986). Research-based tools for bringing about successful school 
improvement. Austin, TX: Southwest Educational Development Laboratory. 
McBrien, Donald E. (1984). Guidance program management model, K12. Towson, 
MD: Baltimore County Public Schools. 

McCaslin, N. L; Gross, Charles U and Walker, Jem' P. (1979). Career education 
measures: A compendium of evaluation instruments. Columbus, OH: National Center 
for Research in Vocational Education. Research and Development Series No. 166. 
McGeever, James M. (1980). Community survey model for school procedural guide. 
Charleston, WV: Appalachia Educational Laboratory, Occasional Paper 008. 
Miller, Juliet; Mastie, Marjorie M. ; and Womer, Frank B. (1977). Career exploration 
student needs assessment final item pool. Wayne, Ml: Wayne Intermediate School 
District. 

Mitchell, Anita M. ( 1978). Ways to evaluate different types of career education 
activities: A handbook of evaluation models. Palo Alto, CA: American Institutes for 
Research. 

Ohio State Department of Education (1984). Guidance services evaluation inventory 
(GSEI). Columbus, OH: author. 

Roberson, E. Wayne ( 1972). Developing observation systems. Tucson, AZ: Educational 
Innovators Press. 

Robie, Brian D.; Gansneder, Bruce M.; and Van Hoose, William H. ( 1979). School 
guidance and counseling program outcomes and measures for their assessment. 
Measurement and Evaluation in Guidance, 12(5). 

Sheldon, Carolyn ( 1987). Guaranteed guidance and counseling services model. 
Portland, OR: Portland Public Schools. 

;>t(K'kard,Jean (1985). Measures of school climate: Needed improvements suggested by a 
review of the organizational literature. Eugene, OR: Center for Educational Policy and 
Management, College of Education, University of Oregon. 

Vetter, Louise and Sechler, Judith (1987). High school career guidance programs that 
work: Seven case studies. Columbus, OH: National Center for Research in Vocational 
Education. 

Walther, Regis H. ( 1975). The measurement of work— relevant attitudes. Washington, 
DC: George Washington University, A Manpower Research Project. 
Worthen, Blaine R. (1987). Curriculum evaluation. La Plata, MD: Charles County Board 
of Education. 



115 



1U 



Follow-up 



Bell, Charles W. ( 1973). Student follow-up system: A conceptualization. Columbus, OH: 
National Center for Research in Vocational Education. 

Department of Occupational Education and Technology, Texas Educational Agency 
( 1979). Tex Sis follow up: Post secondary student follow-up management information 
system. Monograph 9. First Year Graduate Data Summary. Austin, TX: author. 
Fielding, Joyce and Fielding, Marvin (1976). Conducting follow-up and follow through 
programs. Module 4 of National Consortium on Competency -Based Staff Development. 
Palo Alto, CA: American Institutes for Research. 

Franchak, Stephen J. and Spirer, Janet E. ( 1978). Evaluation handbook Volume I: 
Guidelines and practices for following studies of former vocational education students. 
Columbus, OH: Center for Research in Vocational Education. 
Franchak, Stephen J. and Spirer, Janet E. (1979). Evaluation handbook Volume II: 
Guidelines and practices for follow-up studies of special populations. Columbus, OH: 
Center for Research in Vocational Education. 

Harrington, Lois G. ( 1984). Establish a student placement setvice and coordinate follow- 
up studies: Revised edition. Module LT C-4. Columbus, OH: National Center for Research 
in Vocational Education. 

Sears, Susan Jones ( 1980). Follow-up and follow through: Facilitating program leavers 
adjustment and guidance program improvement. Columbus, OH: National Center for 
Research in Vocational Education. 

Sears, Susan Jones ( 1985). Facilitate follow-up and follow through. Columbus, OH: 
National Center for Research in Vocational Education. 



Needs Assessment 

American College Testing ( 1982). River City high school services— a conceptual model. 
Iowa City, IA: author. 

Datta, Lois- Ellin (1981). Evaluation in change: Meeting new government needs. Beverly 
Hills, CA; Sage Publications, Sage Research Progress Series in Evaluation, vol. 12, 

Gray, Sara B. and Perrcello, Patrick J. (1986). Guidance program evaluation. Baltimore, 
MD: Baltimore City Public Schools. 

Johnson, C. D. (1982). Educational and career planning individual guidance 
assessment, Form B (middle/junior) and Form C (high school). Annapolis, MD: Anne 
Arundel County Public Schools. 

Kaufman, Roger (1987). A needs assessment primer. Training and Development 
Journal. Washington, DC; American Staff Training Development. 

Popham, James W. (1972). An evaluation guidebook. Los Angeles, CA: The Instructional 
Objectives Exchange. 



9 

ERLC 



116 

112 



Smith, Thomas and Johnson, C. D. ( 1972). Priority counseling sunvy. Irvine, CA: 
Johnson and Associates. 

Witkins, Ruth Belle ( 197S). An analysis of needs assessment techniques for educational 
planning at state, intermediate, and district lerek Hayward, CA: Office of the Alameda 
County Superintendent of Schools. 



General Reference 

Bloom, Benjamin S. ; Hastings, J. Thomas: and Madaus, George F. ( 1971 ). Handbook on 
formative and summative evaluation of student learning. New York: McGraw-Hill 
Book Co. 

California Asstxiation of Program F.valuators ( 1980, March >7). Beyond survival. 
Proceedings from the 6th Annual Conference. 

Capuzzi, Dave ( 1987). A selected bibliography for personal and social competency: 

Developing skills for the future. Ann Arbor, MI: F.RIC Clearinghouse. 

Cecil, Jean Houchins and Comas, Robert E. ( 1987). Development of strategies for the 

preservation of school counselor preparation programs: A monograph. Alexandria, VA: 

American Association for Counseling and Development: A Publication of SACKS. 

The Foundation Center ( 1980). Conducting evaluations: Three perspectives. New York, 

NY: author. 

Gephart, William J. , Ingle, Robert B.; and Marshall, Frederick J., Fd. ( 1976). Evaluation 
in the affective domain. Phi Delta Kappa. 

Glaser, Barney G.s and Strauss, Anselm L. ( 1967). The discovery of grounded theory: 
Strategies for qualitative research. Hawthorne, NY: Aldinc Publishing Co. 
Hotchkiss, Lawrence and Dorsten, Linda F. ( 1985). Outcomes of career guidance and 
counseling in high school. Columbus, OH: National Center for Research in Vixational 
Education. 

Hotchkiss, Lawrence and Vetter, Louise (1987). Outcomes of career guidance and 
counseling. Columbus, OH: National Center for Research in Vocational Education. 
Jung, Steven M. (1977). Evaluating the work education consortium project: An overview 
of issues and options. Washington, DC: National Manpower Institute. 
Kaufman, Roger and Thomas, Susan ( 1980). /-valuation without fear. New York: New 
Viewpoints, a division of Franklin Watts. 

Kaufman, Roger and Stone, Bruce ( 1983). Planning for organizational success. New 
York, NY: John Wiley and Sons. 

Krathwohl, David R. (1966). How to prepare a research proposal: Suggestions for those 
seeking funds for behavioral science research. Syracuse, NY: Syracuse University 
Bookstore. 

Landon, Glenda I. ( 1983). Counseling and guidance program analysis handbook 
Madison, Wl: School Evaluation Consortium, University of Wisconsin. 
Marmachev, Helen L ( 1979). Program evaluation and accountability Ann Arbor, MI: 
ERIC Counseling and Personnel Services Clearinghouse. 



ERIC 



117 

1 J 3 



McAuliffe, Garrett J. (1988). Outcomes of a group career planning process: Occasional 
paper no, 126. Columbus, OH: National Center for Research in Vocational Education. 
Metfcssel, M. S. and Hammond, M. D. (undated). How to develop accountability 
programs in the affective domain Los Angeles, CA: University of Southern California. 
Miller, Lawrence R. ( 1974). A peek at perk: A brief introduction to program evaluation 
and review technique. Escondido, CA: The Center for Leadership Studies. 

Mitchell, Anita M. ( 1981). Practical evaluative research for practitioners. Denver, CO: 
Love Publishing Co. 

Mitchell, Anita M. (1984). Guidelines for too early childbearing programs. Sponsored by 
the Charles Stewart Mott Foundation. Los Alamitos, CA: Southwest Regional Laboratory. 
Noll, Victor H. and Scanned, Dale P. ( 1972). Introduction to Educational Measurement- 
Third Edition Boston, MA: Houghton Mifflin Co. 

Popham, James W. (1967). Simplified Designs for School Research. Inglewood, CA: 
Southwest Regional Laboratory. 

Rhoadcs, David J. ( 1982). Defining school counselor competencies: More questions 
than answers? Maryland Personnel and Guidance Association Journal. Baltimore, MD. 
Riggs, Ron C. ( 1978) Evaluation of counselor effectiveness, Ann Arbor, MI: ERIC 
Counseling and Personnel Services Clearinghouse. 

Sax, Gilbert ( 1974). Principles of education measurement and evaluation. Belmont, 
CA: Wadsworth Publishing Co. 

Sheldon, Carolyn (undated). Guaranteed guidance and counseling services model. 
Portland, OR: Portland School District No. 1. 

Starr, Harold (undated). A framework for implementing critical evaluation and 
planning Columbus, OH: National Center for Research in Vocational Education. 
Stufflebeam, Daniel L; Foley, Walter J.; Gephart, William J.; Guba, Egon G.; Hammond, 
Robert L; Merriman, Howard O.; and Provus, Malcolm M. (1971). Educational 
evaluation and decision making. Itasca, IL: F. E. Peaccxrk Publishers, Inc. 
Tallmadgc, G. Hasten (undated). Avoiding problems in evaluation. Mountain View, CA: 
RMC Research Corp. 

Thorndike, Robert L. and Hagen, Elizabeth (1955V Measurement and evaluation in 

psychology and education. New York: John Wiley and Sons, Inc. 

Watkins, Richard W. and Jenks, C. Lynn ( 1976). Values and information: An approach 

to evaluation planning San Francisco, CA: Far West laboratory. 

Webb, Eugene J.; Campbell, Donald T.; Schwartz, Richard D.; and Scchrest, Lee ( 1966). 

Unobtrusive measures: Nonreactive research in the social sciences. Chicago, IL: Rand 

McNally College Publishing Co. 

Wcllmann, Frank E. (1970). Evaluation of vocational guidance— local level. Columbia, 
MO: University of Missouri. 

Wholey Joseph S. ( 1979). Evaluation: Promise and performance. Washington, DC: The 
Urban Institute. 

Witkin, Belle Ruth and Stephens, Kent G. ( 1973). Eault tree analysis: A management 
science technical for educational planning and evaluation. Hayward, CA: Alameda 
County School Department of Education. 



The Contributors 



John O. Crites 

John O. Crites is Director of the Crites Career Centers in Boulder, Colorado. He also 
serves as a consultant to universities, school districts, and business and industry 
throughout the United States. He has previously held appointments as Professor of 
Psychology and Director of the University Counseling Center at the University of Iowa, 
Professor of Psychology at the University of Maryland, and Research Professor at Kent 
State University. He received his B.A. at Princeton University (1950) and his Ph.D. in 
counseling psychology at Columbia University (1957). 

Crites' published works include: Vocational Psychology': Career Counseling Models, 
Methods and Materials as well as the Career Maturity Inventor)'. He has also 
contributed chapters to numerous volumes including each of the National Vcxational 
Guidance Association's commemorative volumes. He has received the American 
Association of Counseling and Development Research Award for his Psychological 
Monograph and the Eminent Career Award from the National Career Development 
AsscKiation. 



Harry N. Drier 

Harry N. Drier is currently President of the Career Education and Training Asscxiates 
Corp. and Senior Scientist Emeritus at Ohio State University, the Center on Education 
and Training for Employment. Drier earned a B.S. in education at Drake University, an 
M.S. at the University of Wisconsin, Oshkosh in guidance and counseling, and an Ed.S. 
in educational administration at the University of Wisconsin, Madison. He has authored 
several books, numerous chapters in guidance texts, and over thirty journal articles. 

Prior to his research and development work at Ohio State, he served as a State Guidance 
Supervisor at the Wisconsin Department of Public Instruction, and as a Uxal school 
counselor and guidance director. Professionally, he has served as President of the NVGA 
(now the National Career Development Association), the Guidance Division of the 
American Vocational Association, and the National Vocational and Technical Honor 
Society. 



9 

ERIC 



119 



■5 



Norman C. Gysbers 



Norman C. Gysbers is Professor of Educational and Gumseling Psychology at the 
University of Missouri, Columbia. He received his BA ( 19^4) from Hope College and 
his MA (1959) and Ph.D. ( 1963) from the University of Michigan. 

Gysbers was Editor of the Vocational Guidance Quarterly, 1962 ^0; President of the 
NCDA. 19^2: President of the AACD (formerly the American Personnel and Guidance 
Vwational Association), 19"9*82. Currently he is the editor of the Journal of Career 
Development Since 196" 7 he has served as director of numerous national and state 
projects on career development and career counseling and school guidance program 
development, implementation, and evaluation. 



CO. "Curly" Johnson 

After many years in the field as a guidance coordinator and college instructor, C. D. 
"Curly" Johnson is currently President of Johnson and Asscxiates, Leadership 
Consultants of Southern California. He is known for workshops, consultation, and 
presentations on a wide range of topics including leadership styles, planning models, 
organizational elements, motivation, management theories and strategies, 
communication, team building, personal power, use of technology' in humanistic 
organizations, critical thinking, listening skills, and program evaluation. Johnson 
received his Ph D. in leadership and human behavior from the United States 
International University, San Diego, his M.S. from the University of Southern California, 
and his BA from Portland State University. Oregon. Publications include five books and 
numerous articles, manuals, evaluation reports, and standardized assessment 
instruments. 



Sharon K. Johnson 

Sharon K* Johnson is currently an Associate Professor in the Division of Educational 
Administration and Counseling at California State University* of Los Angeles. She earned 
her diKtorate in counselor education from Virginia Polytechnic Institute and State 
I Yiiversity, her matter's degree in counseling from California State University at 
Fullerton. and her bachelor of science from the University' of California at Los Angeles. 
Johnson has been an elementary, middle, and high school teacher, a secondary schcx)l 
counselor in California, and Director of Pupil Services in Maryland. She has written 
numerous articles and has consulted with many educational and business organizations 
in the areas of guidance program development and evaluation, management training, 
marriage family child counseling, and career development. 



9 

ERLC 



Edwin A Whitfield 



Edwin A. "Whitfield has been a secondary school teacher in Iowa, a counselor in 
Ohio, a county guidance coordinator in California, and a counselor educator in 
California and Kansas. Currently he is the State Supervisor of Guidance and Counseling 
for the Ohio Department of Education. Whitfield has worked as a consultant for the 
U.S. Department of Labor, state education departments, community agencies, and 
numerous school districts throughout the United States. He has served as president of 
the NVGA, as editor of the Vocational Guidance Quarterly, and on numerous AACD and 
NCDA committees. He received his master's degree from Ohio University and his Ph.D. 
from the University of Iowa. 



9 

ERLC 



1 17 

121 



\ 



The Who, What, Where, When, 
How, and Why of Evaluating 
Guidance Programs 



r 



This guide was developed through a cooperative effort of 
American College Testing and The National Consortium 
'of State, Career Guidance Supervisors. 



♦ 

ERIC 




» I • \ \ T K >\ \ 
V \ { (>\SOKIIt VI 



BEST COPY AVAILABLE 




ISBN .l-5bDm-Dll-l 



» 




